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ABSTRACT 


The  Calgary  Board  of  Education  and  the  Calgary  Catholic  Board  of 
Education  have  developed  indicators  of  quality  student  performance  for  the 
assessment  of  school  art  composition  and  mathematical  problem  solving. 
Practising  teachers  and  school  system  specialists  in  art  and  mathematics  have 
developed  and  classroom-tested  materials  to  identify  and  document  quality 
indicators.  The  resulting  qualitative  materials  describe  a performance 
assessment  process  which  celebrates  diversity  in  student  responses.  The 
process  depends  on  informed  professional  judgement,  shared  exemplars  and 
vocabulary,  and  student  involvement  in  self-assessment. 

This  report  describes  the  beginning  and  development  of  the  project,  the 
rationale  and  literature  that  informed  the  work,  and  the  apparent  impact  of 
the  project,  as  the  project  evolved  from  September  1989  to  May  1992.  The 
project  results  to  date  indicate  that  performance  assessment  has  positive 
effects  on  learning  and  teaching,  and  that  further  explorations  and 
applications  of  the  quality  indicators  project  would  be  educationally 
significant. 
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CHAPTER  1:  INTRODUCTION 


This  chapter  describes  the  background  and  purposes  of  the  Calgary 
Educational  Quality  Indicators  (EQI)  Project,  along  with  the  assumptions, 
definitions,  and  methodologies  that  characterize  the  project  activities. 

Background 

In  the  Spring  of  1989,  officials  from  the  Calgary  Board  of  Education 
(CBE),  the  Calgary  Catholic  Board  of  Education  (CCBE)  and  Alberta 
Education  negotiated  a contract  as  part  of  the  provincial  EQI  Initiative 
that  committed  the  parties  to  a collaborative  project  entitled  "The 
Development  of  Indicators  of  Academic  Excellence"  (called  the  EQI 
Project  or  the  "project"  in  this  report).  The  purpose  of  the  project  was  to 
develop  and  pilot  test  indicators  of  quality  student  work  in  areas  of  school 
art  and  mathematics.  Personnel  were  identified  and  assigned  to  the  project 
at  the  beginning  of  the  1989  - 90  school  year. 

The  Alberta  art  curriculum  includes  depiction  and  composition  from  grades 
1 through  12.  Art  is  a compulsory  course  for  students  only  in  grades  1 to 
6.  Art  achievement  is  often  assessed  in  terms  of  student  effort,  attitude, 
participation  or  independent  teacher  professional  judgement.  There  are  no 
assessment  devices  available  that  were  based  on  commonly  understood 
expert  standards  regarding  the  quality  of  student  artwork. 

The  articulation  and  communication  of  quality  indicators  in  student 
artwork  is  intended  to  result  in  improved  teaching  behaviours  and 
improved  student  learning.  The  project  provides  documentation  of  those 
observable  qualities  in  student  art  which  should  be  recognized  and 
encouraged  and  which  assist  in  the  development  of  teacher  expertise  in 
the  area  of  art. 

Problem  solving  was  selected  for  the  development  of  quality  indicators 
because  of  the  importance  of  problem-solving  skills  in  the  mathematics 
program  and  because  of  a predicted  transfer  of  the  problem-solving 
practices  to  other  subject  areas.  Problem  solving  in  mathematics  is  both  a 
specific  content  strand  as  well  as  a unifying  construct  for  the  entire 
program.  This  is  evident  in  Alberta  Education  curriculum  documents  for 
elementary,  junior  high,  and  senior  high  school  mathematics  courses.  The 
assessment  of  problem  solving  involves  the  examination  of  the  thinking 
processes  that  underlie  student  answers  to  problems. 

Traditional  assessment  practice  in  mathematics  has  been  most  concerned 
with  the  product  or  answer  given.  Mathematics  teachers  and  curriculum 
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experts  encourage  the  assessment  of  problem-solving  processes,  but  few 
materials  were  available  to  all  teachers  to  assist  in  this  task. 

Context 

Calgary  is  a city  of  700,000  people  located  100  kilometres  east  of  the 
Rocky  Mountains  in  central  Alberta.  There  are  two  publicly-supported 
school  systems  to  serve  the  urban  area:  the  Calgary  Board  of  Education, 
with  an  enrolment  of  over  94,000  students  and  the  Calgary  Catholic 
Board  of  Education,  with  an  enrolment  of  about  30,000  students.  The 
CBE  and  the  CCBE  are  respectively  the  third  largest  and  the  twenty-fifth 
largest  school  systems  in  the  country  (Canadian  Education  Association, 
1992).  Both  school  systems  provide  subject  area  assistance  to  classroom 
teachers  through  central  office  specialists  in  most  school  subject  areas. 
These  specialists  are  involved  in  the  organization  and  delivery  of  teacher 
inservice  programs,  the  selection  and  development  of  materials  to  support 
teaching  and  assessment,  and  the  recommendation  of  school  system 
policies  that  affect  their  subject  specialties.  There  is  a long  history  of 
professional  collaboration  between  the  two  school  boards. 

Rationale 

The  CBE  and  CCBE  are  committed  to  discovering  and  using  the  best 
possible  ways  to  gather  information  about  and  report  on  student 
achievement  in  their  schools.  The  EQI  Project  provided  an  opportunity  to 
emphasize  this  commitment. 

The  CBE  and  the  CCBE  report  on  the  achievement  of  individual  students 
and  of  groups  of  students  using  a variety  of  techniques  and  materials. 
Some  measures  and  reports  about  student  achievement,  such  as  the 
Student  Achievement  Tests  and  Diploma  Examinations,  are  prepared  and 
distributed  by  Alberta  Education.  Other  information  regarding  student 
achievement  is  prepared  and  distributed  by  school  system  officials,  by 
school  administrators  and  by  classroom  teachers. 

Most  of  the  reports  about  student  achievement  are  in  the  form  of  results  on 
paper-and-pencil  tests  where  there  is  a single  correct  answer.  Student 
performances  which  are  creative  or  original  escape  inclusion  in  traditional 
tests,  although  they  are  powerful  indicators  of  student  success.  Student 
achievements  in  creative  expression,  critical  thinking,  and  problem  solving 
fall  into  this  performance  category.  These  complex  student  performances 
are  difficult  to  report  on  using  numbers,  averages,  and  percentiles. 
Professional  educators  require  new  alternatives  for  assessing  and  reporting 
on  student  achievement  that  is  demonstrated  in  complex  performances. 
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This  project  addresses  the  lack  of  such  alternatives  through  the 
development  of  quality  indicators  in  art  and  mathematics. 

Teachers  and  fine  arts  specialists  in  the  CCBE  had  identified  a need  for 
assessment  materials  for  the  visual  arts.  They  requested  help  with  the  task 
of  identifying  quality  in  the  art  works  that  students  produce.  This  request 
formed  the  basis  of  one  part  of  the  EQI  Project  proposal.  The  contract 
developers  were  also  aware  of  requests  for  assessment  materials  for 
problem  solving  in  mathematics.  Problem  solving  has  recently  become  a 
very  important  element  of  mathematics  courses  throughout  the  grades,  and 
teachers  and  students  needed  new  materials  to  help  demonstrate  student 
achievement  in  this  content  area. 

Purposes 

The  purposes  of  the  EQI  Project  were  the  following: 

• to  identify  indicators  of  quality  work  which  can  be  applied  in  the 
assessment  of  student  achievement  in  areas  of  art  and  mathematics, 

• to  identify  conditions  that  appear  to  enhance  or  limit  quality 
achievement, 

• to  develop  strategies  for  collecting  quality  student  work  which 
include  recorded  interviews  with  teachers  and  students,  and 

• to  examine  quality  work  with  emphasis  on  alternative  assessment 
strategies  to  paper  and  pencil  testing. 

Assumptions 

EQI  project  advisors  and  practising  teachers  appeared  to  have  common 
understandings  or  assumptions  about  learning,  teaching  and  assessment 
that  served  as  a background  from  which  to  proceed  with  the  project  work. 
These  three  elements  organize  our  assumptions. 

About  Learning 

There  is  a difference  between  information  and  knowledge,  although 
students  attain  both  as  a result  of  schooling.  Information  refers  to  the 
content  of  the  school  curriculum  that  can  be  memorized  and  recalled  for 
presentation.  Knowledge,  on  the  other  hand,  is  the  result  of  a student’s 
making  meaning  out  of  information  and  experience.  The  possession  of 
knowledge  is  demonstrated  in  the  performance  of  tasks  that  challenge 
students  to  apply  what  they  know  to  a new  situation. 
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About  Teaching 


Teachers  are  responsible  for  providing  students  with  learning  experiences 
that  assist  students  in  making  meaning  and  in  practising  their  learned  skills. 
The  teacher  is  like  a coach  or  mentor. 

Teachers  can  identify  quality  in  student  work,  although  they  may  not  all 
be  able  to  articulate  the  characteristics  that  indicate  quality,  or 
communicate  about  these  characteristics  in  the  same  way. 

About  Assessment 

Traditional  paper-and-pencil  tests  which  require  simple,  single  answers  to 
highly  structured  situations  cannot  adequately  assess  student  knowledge. 
Assessment  practice  has  an  impact  on  teaching  practice. 

Student  performances  include  some  unexpected  responses.  Teachers  need 
to  be  able  to  analyze  these  responses,  place  them  in  context,  and  comment 
appropriately.  Practising  expert  teachers  can  recognize  qualitative 
differences  among  student  performances. 

Teachers  and  students  can  develop  skills  in  ’’connoisseurship”  regarding 
the  performance  of  complex  tasks.  That  is,  they  can  develop  *'a  highly 
differentiated  array  of  anticipatory  schema  that  enable  one  to  discern 
qualities  and  relationships  that  others  ...  are  less  likely  to  see”  (Eisner,  1985, 
p.  153).  Samples  of  excellent  performances  can  be  used  effectively  to  help 
develop  connoisseurship  skills. 

Learners  develop  appropriate  self-assessment  skills  with  coaching  and 
practice. 


Definitions 

In  this  project  quality  indicators  are  defined  as  observable 
characteristics  of  excellence  established  through  consensus  of 
professional  judgement  among  practising  teachers. 

Assessment  is  gathering  information  or  observable  evidence  of  what  a 
learner  can  do.  Evaluation  is  making  judgements  based  on  interpreting 
assessment  information.  Assessment  and  evaluation  are  part  of  the 
teaching  process. 
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Design 


This  project  was  designed  as  action  research:  we  began  with  the 
experience  of  actual  practice,  and  reviewed  this  experience  to  develop 
frameworks  and  proposals  for  further  study.  In  this  section,  we  briefly 
describe  activities,  structure  and  evaluation  as  these  were  shown  in 
project  design. 

Activities 

The  project  design  incorporated  three  phases:  development,  field  testing, 
and  pilot  testing,  which  correspond  to  the  three  years  of  project  funding. 
Activities  in  the  developmental  phase  included  the  review  of  the  literature, 
establishment  of  committees,  and  preparation  of  initial  structures  for 
indicators  of  quality  student  achievement.  Committees  of  teachers  and 
subject  specialists  reviewed  student  work,  identified  characteristics  of 
quadity  work,  considered  alternative  approaches  for  recording  assessment 
information,  and  proposed  activities  and  materials  for  use  in  classrooms. 

In  the  field-testing  phase,  project  personnel  tried  a variety  of  materials  and 
approaches  in  classrooms.  Promising  approaches  were  pursued  and 
evidence  of  the  results  of  field  tests  was  collected.  This  evidence  was  used 
to  revise  the  materials  and  to  structure  plans  for  pilot  testing. 

Pilot  tests  were  carried  out  by  teachers  who  had  no  previous  experience 
with  the  EQI  project.  The  pilot  teachers  tried  out  the  assessment  materials 
during  their  regular  teaching  practice  and  during  specifically  designed 
classroom  activities.  Project  advisors  provided  inservice  prior  to  piloting 
and  support  to  pilot  teachers  during  the  pilot  period. 

Structure 

Art  and  mathematics  subject  specialists  from  the  two  school  systems 
provided  advice,  support  and  direction  for  the  project.  These  specialists 
were  the  advisors  to  the  project  as  well  as  participants  in  project  activities. 

Classroom  teachers,  identified  by  subject  specialists  for  their  expert 
classroom  practice  and  professional  commitment,  provided  the 
developmental  work,  the  testing  of  materials  in  classrooms,  and  thoughtful 
reflections  and  evaluations.  School  principals  approved  each  teacher’s 
participation  and  then  the  teachers  volunteered  to  participate.  The  first 
team  of  teachers  was  asked  to  work  with  the  project  for  the  first  two  years. 
Because  much  of  their  work  was  action  research  into  their  own  practice, 
they  are  called  ’’teacher  researchers.”  Teachers  whose  primary  role  was  to 
try  out  specific  materials  and  practices  in  their  classrooms  are  called  ’’field 
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test  teachers"  and  "pilot  test  teachers."  For  every  project  task,  the  project 
teachers  represented  both  the  selected  grade  levels  of  the  study  and  the 
two  school  jurisdictions. 

Senior  superintendents  and  education  officials  served  on  project  steering 
committees  that  provided  coordination  between  the  two  school 
jurisdictions  and  Alberta  Education,  and  provided  direction  to  the  project 
manager.  An  advisory  committee  provid^  a communications  link  to  other 
educational  stakeholders.  Members  of  the  advisory  committee  included: 
university  professors  in  art  and  in  mathematics,  leaders  in  the  art  and 
mathematics  communities,  parent  representatives,  teachers’  association 
representatives,  and  members  of  the  business  community. 

A local  advisory  committee  — consisting  of  representatives  from  the 
University  of  Calgary,  the  Alberta  Teachers’  Association,  parent  councils  in 
both  school  boards,  the  arts  and  mathematics  communities,  and  the 
business  community  in  Calgary  — assisted  in  providing  advice  and 
feedback  regularly  throughout  the  project.  The  project  personnel  who 
formed  the  development,  advisory  and  steering  committees  for  the  project 
are  listed  in  Appendix  A. 

Evaluation 

Project  evaluation  was  both  formative  and  summative  throughout  the 
project.  The  project  evaluation  plan  was  based  on  a "Context,  Inputs, 
Process  and  Product"  (CIPP)  framework  (Stufflebeam  & Webster,  1988). 
This  framework  structures  evaluation  questions  around  the  four  issues:  the 
contextual  environment  for  the  project,  the  input  resources  that  were 
required  and/or  used  in  the  project,  the  processes  and  activities  that  shaped 
the  project,  and  finally,  the  products  of  the  project  and  their  perceived 
usefulness  and  value.  Teachers  and  advisors  regularly  considered 
questions  about  the  processes  and  products  of  the  project.  Their 
recommendations  were  used  to  make  changes  to  activities  and  materials 
throughout  the  project. 

Both  quantitative  and  qualitative  data  were  gathered  about  the  effects  of 
the  project  in  classrooms.  Many  different  information- gathering  methods 
were  used:  students  were  videotaped  and  audiotaped  during  problem- 
solving and  art-making  sessions;  written  responses  were  requested;  survey 
data  were  collected;  and  an  external  interviewer  spoke  with  advisors. 
Quantitative  data  were  collected  regarding:  teacher  behaviours  and  beliefs 
about  art  and  problem  solving;  student  attitudes,  dispositions  and  scores 
on  problem  solving;  and  teacher  reports  of  practice  and  the  impact  of 
EQI-Math  on  their  teaching.  Qualitative  data  were  collected  regarding: 
teacher  beliefs  and  practices  in  art  and  mathematics,  before  and  after 
piloting;  student  responses  to  EQI  materials;  teacher  and  student  journals 
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in  art;  anecdotal  records  of  student  problem  solving;  descriptions  of  the 
mathematics  classroom  environments;  changes  reported  by  teachers  as  a 
result  of  using  EQI  materials;  and  impressions  of  the  project  formed  by 
project  advisors. 

An  external  consulting  agency  completed  a formal  evaluation  of  project 
utility,  management,  and  impact.  This  evaluation  is  found  in  Appendix  D. 

In  this  report,  data  from  more  than  one  source  have  been  used  to  develop 
conclusions  and  recommendations.  The  accumulated  data  have  not  all 
been  analyzed  at  this  time.  However,  original  data  have  been  maintained 
in  project  archives  for  future  reference. 

Scope 

The  art  component  of  the  project  (EQI-Art)  focused  on  the  depiction  and 
composition  components  of  the  Alberta  Education  art  curriculum  insofar  as 
these  are  displayed  in  completed  student  art  works.  Student  art  works 
from  grades  2 and  5 are  the  elementary  exemplars.  At  the  junior  and  senior 
high  school  levels,  art  works  reflect  all  grade  levels,  because  many  schools 
offer  only  one  art  course  or  offer  courses  for  students  from  any  of  the  three 
grades  in  that  division. 

EQI-Math  focused  on  problem  solving,  which  is  both  a content  area  and  a 
method  of  teaching  in  the  Alberta  Education  mathematics  curricula.  Grades 
2,  5,  8,  and  10  were  selected  to  provide  a representative  sample  of  student 
achievement  for  the  EQI  Project.  Grades  2,  5,  and  8 are  midway  through 
each  of  the  school  levels  — primary,  late  elementary,  and  junior  high. 

Grade  10  was  identified  in  place  of  grade  1 1 because  there  was  a new 
curriculum  implementation  affecting  grade  1 1 mathematics  courses  during 
the  EQI  Project  years. 


Limitations 

School  system  specialists  identified  successful,  professionally  involved 
practising  teachers  for  the  EQI  Project.  School  principals  approved  each 
teacher’s  participation  and  then  the  teachers  volunteered  to  participate. 

Twenty-six  practising  teachers  from  the  CBE  and  the  CCBE  were  teacher 
researchers  during  the  initial  phase  of  the  project.  Another  24  teachers 
field  tested  or  pilot  tested  project  materials  in  their  classrooms.  These  50 
teachers  represent  a very  small  percentage  of  the  more  than  5,000 
classroom  teachers  in  the  two  school  systems. 
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Teachers  piloted  the  project  materials  in  16  schools  during  the  first  half  of 
the  1991-92  school  year.  Two  classes  at  each  grade  level  pilot  tested  the 
EQI-Art  and  EQI-Math  materials.  This  is  a very  small  pilot  test,  limited  by 
the  resources  available  for  the  project.  The  project  teachers  were  selected 
because  of  their  expertise  and  commitment  to  professional  development. 
This  fact  may  limit  the  transfer  of  experiences  from  these  teachers  to  the 
population  of  teachers  in  general. 

Overview 

This  chapter  has  described  the  background  to  the  Calgary  EQI  Project,  its 
purposes,  the  organization  and  scope  of  the  project.  Chapter  2 includes  a 
discussion  of  the  related  literature  and  concludes  with  the  implications  of 
the  literature  for  the  structure  and  activities  of  the  EQI  Project.  Chapters  3 
and  4 discuss  the  activities  and  findings  from  the  EQI-Art  and  the 
EQI-Math  projects  respectively.  Chapter  5 brings  the  two  projects 
together  with  a discussion  of  the  findings  of  the  project  as  a whole  and 
recommendations  for  future  activities. 
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CHAPTER  2:  RELATED  LITERATURE 


The  teachers  and  advisors  working  on  the  EQI  Project  began  their  work 
based  on  their  personal  practical  knowledge  of  schools  and  schooling. 

The  project  coordinator  responded  to  and  supported  the  committee 
deliberations  by  compiling  literature  of  interest  to  the  committees  in  the 
areas  of  learning,  teaching  and  assessment.  The  following  discussion 
incorporates  the  key  aspects  of  that  literature,  and  concludes  with 
implications  for  the  EQI  Project. 

Learning 

The  EQI  Project  developed  new  materials  for  the  assessment  of  student 
learning  in  areas  of  art  and  mathematics.  The  understanding  about 
learning  is  an  important  basis  for  the  project's  work.  This  section  addresses 
this  understanding  in  two  subsections:  1)  from  the  perspective  of  goals  of 
education  and  schooling,  and  2)  as  part  of  the  view  of  learning  in 
cognitive  theory. 

Goals  of  Education  and  Schooling 

The  aim  of  education  in  Alberta  is  to  develop  the  knowledge,  the  skills,  and 
the  positive  attitudes  of  individuals,  so  that  they  will  be  self-confident, 
capable  and  committed  to  setting  goals,  making  informed  choices  and 
acting  in  ways  that  will  improve  their  own  lives  and  the  life  of  their 
community  (Alberta  Education,  1991). 

The  learner  is  expected  not  only  to  have  knowledge  but  also  be  able  to 
use  knowledge  in  significant  ways.  "What  matters  is  not  that  one  can  read, 
but  rather  that  one  does  read  and  that  what  one  reads  is  worth  reading" 
(Eisner,  1983,  p.  49).  Alberta's  goals  of  secondary  education  specify  that 
schools  will  assist  students  to  accumulate  a wide  variety  of  knowledge  and 
skills,  and  also  intend  that  students: 

• develop  the  ability  to  think  conceptually,  critically  and  creatively;  to 
acquire  and  apply  problem-solving  skills;  to  apply  principles  of  logic; 
and  to  use  different  modes  of  inquiry; 

• assume  increasing  responsibility  for  independent  and  continuous 
learning,  and  develop  positive  attitudes  toward  learning  while  in 
school,  in  preparation  for  self-directed,  lifelong  educational 
experiences  (Alberta  Education,  1991,  p.  4). 
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Educators  and  others  often  use  the  term  "literacy"  in  defining  the  goals  of 
education.  Discussions  about  literacy  reiterate  the  message  that  the 
possession  of  "basic"  skills  is  insufficient.  Resnick  and  Klopfer  (1989) 
extend  literacy  into  "competent  thinking."  For  the  Calgary  Board  of 
Education  (1989),  literacy  is  a necessary  condition  for  personal  growth, 
including  the  abilities  people  have  to  understand  and  to  participate  and 
communicate  in  the  communities  in  which  they  find  themselves.  Greene 
(1983,  1990)  speaks  of  literacy  as  an  ability  to  extend  personal  power  over 
symbols  and  knowledge  in  a whole  range  of  intellectu^  environments,  the 
arts  and  technologies,  literature,  politics,  and  social  relationships. 

Definitions  of  mathematical  hteracy  imply  a similar  set  of  objectives, 
extending  from  the  development  of  technical  mathematical  proficiency  to 
the  ability  to  understand  and  appreciate  the  contributions  of  mathematical 
concepts  to  everyday  decision  making,  to  culture  and  to  civilization 
(National  Council  of  Teachers  of  Mathematics,  1989). 

Cognitive  Theory 

Currently,  learner’s  thinking  skills  have  become  a focus  of  research  study 
in  education.  Modem  cognitive  theory  asserts  that  "people  are  not 
recorders  of  information  but  builders  of  knowledge  structures"  (Resnick  & 
Klopfer,  1989,  p.  4).  Learning  is  a complex,  meaning-making  activity  that 
can  have  many  representations,  depending  on  the  skills,  experience  and 
environment  of  the  learner.  Both  art  making  and  mathematical  problem 
solving  are  depicted  as  this  kind  of  learning.  Visual  art  is  the  observable 
result  of  cognitive  decision  making,  and  its  production  combines 
perception,  use  of  visual  symbol  systems,  technical  skill,  cultural 
appreciation,  and  emotion  (MacLeod,  1989).  Essays  on  the  processes  of 
problem  solving  (Charles  & Lester,  1987;  National  Council  of  Teachers  of 
Mathematics,  1989;  Schoenfeld,  1989)  emphasize  the  complexity  of  the 
process,  including  logical  cognitive  processes  as  well  as  collaboration  with 
others,  dispositional  factors,  and  other  environmental  interactions  in  the 
depiction  of  problem-solving  skills  and  processes. 

Cognitive  theorists  and  educators  endeavour  to  understand  what  is 
happening  in  the  mind  of  the  learner.  Norris  (1989)  suggests  that  teachers 
need  to  take  the  extra  time  to  go  beyond  the  student  answer  and  to 
encourage  critical  thinking.  He  points  out  the  usefulness  of  talking  to  test- 
takers,  encouraging  them  to  reflect  and  reason  about  their  answers. 
Gardner  (1991)  encourages  the  development  of  expert  understanding  in 
learners,  rather  than  settling  for  "correct-answer  compromises"  which  may 
cover  up  a lack  of  significant  understanding.  Skemp  (1989)  considers  the 
difference  between  teaching  for  "instrumental  understanding"  which 
expects  that  students  have  formulas  or  habits  of  thought,  and  teaching  for 
"relational  understanding,"  which  requires  that  students  develop 
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conceptual  structures  to  guide  thinking  about  mathematical  concepts.  He 
calls  on  teachers  to  first  understand  concepts  children  hold,  and  then  to 
help  them  build  new,  more  appropriate  schema. 

Metacognition,  or  how  students  evaluate  and  control  their  own  thinking  is 
expected  to  be  systematically  developed  in  schools  (Romberg  & 

Carpenter,  1986;  Schoenfeld,  1989).  Snow  (1989)  provides  a model  for 
research  on  assessment  that  includes  these  metacognitive  processes  and 
recommends  assessment  models  that  capture  higher  order  cognitive  skills 
and  conative  structures.  Processes  of  metacognition  engage  the  student  in 
self-assessment  and  self-regulation.  Schoenfeld  (1989)  describes  this 
engagement  as  part  of  that  understanding  that  leads  to  mastery.  The  major 
goal  in  mathematics  should  be  teaching  "power"  or  higher  order  thinking 
skills.  He  encourages  the  use  of  coaching  techniques  when  working  with 
students  on  problem  solving  in  the  belief  that  students  can  be  taught  to 
regulate  their  own  thinking  through  metacognition. 

Teaching 

The  teacher  is  the  critical  factor  according  to  much  of  this  research,  which 
calls  for  the  teacher  to  arrange  learning  environments  and  activities  which 
"stimulate  and  nourish  students'  own  mental  elaborations  of  knowledge 
and  to  help  them  grow  in  their  capacity  to  monitor  and  guide  their  own 
learning  and  thinking"  (Resnick  & Klopfer,  1989,  p.  5).  In  this  view  of 
learning,  teacher  expertise  becomes  increasingly  critical  to  student 
achievement.  This  section  addresses  how  the  conceptions  of  teaching 
have  changed  to  value  the  teacher  as  reflective  practitioner  and  as 
mentor  and  coach. 

Reflective  Practitioner 

As  expectations  of  education  are  sorted  out  in  the  political  process,  and  as 
research  provides  information  about  relationships  within  the  education 
process,  paradigms  or  persistent  pictures  emerge  of  the  educational  reality 
that  change  only  gradually  over  time.  Shavelson  (1988)  and  Brophy  and 
Good  (1986)  document  three  paradigms  that  have  been  used  to 
understand  the  teaching  process:  the  "trait  conception"  of  good  teaching 
which  connected  performance  with  the  appearance  of  personal 
characteristics  in  the  teacher  in  the  1940s  and  1950s;  the  "skills 
conception"  of  good  teaching  which  focused  on  the  microteaching  skills 
and  a repertoire  of  competencies  for  instruction  in  the  1960s  and  1970s; 
and  the  "reflective  practitioner"  conception  of  teacher  that  places  the 
teacher  as  mediator  in  the  process  of  student  construction  of  knowledge  in 
the  1980s  and  now  into  the  1990s. 
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Clark  and  Peterson  (1986)  describe  the  emergence  of  the  view  of  the 
teacher  as  reflective  practitioner  to  replace  the  notion  of  teacher  as  skilful 
technician.  This  view  is  consistent  with  the  teacher  as  clinician  and  action 
researcher,  which  is  evident  in  the  work  of  Duckworth  (1986),  Goodlad 
(1986, 1987),  Clandinin  (1986),  Calfee  and  ffiebert  (1988),  Costa  (1989), 
Shavelson  and  Stem  (1981),  Sirotnik  and  Clark  (1988).  Teacher  thinking, 
as  conceptualized  in  this  body  of  research,  is  a central  set  of  moderating 
contextual  factors  that  influence  curriculum  effectiveness,  school 
effectiveness,  and  student  achievement.  Teacher  thinking  must  be  the 
focus  of  teacher  development  practices  at  all  levels  in  education. 

If  we  wish  to  develop  intelligent  behaviour  as  a significant 
outcome  of  education,  instmctional  strategies  purposefully 
intended  to  develop  children's  metacognitive  abilities  must  be 
infused  into  our  teaching  methods,  staff  development  and 
supervisory  processes  (Costa,  1984,  p.  58). 

Mentor  and  Coach 

Duckworth  (1986)  encourages  teachers  to  put  students  in  direct  contact 
with  phenomena  to  be  studied  and  to  understand  the  sense  students  are 
making.  She  suggests  that  mentoring  and  coaching  students,  as  a partner 
in  their  learning,  has  important,  very  desirable  results.  First,  students  gain 
clarity  of  thought  by  explaining  themselves  to  others.  Second,  they 
determine  for  themselves  what  they  want  to  learn.  Third,  students  come  to 
depend  on  themselves,  they  become  the  judges  of  what  they  know  and 
believe.  Fourth,  students  recognize  the  powerful  experience  of  having 
their  ideas  taken  seriously.  Fifth,  students  learn  an  enormous  amount  from 
one  another.  In  this  process,  learners  come  to  recognize  knowledge  as  a 
human  product,  and  to  be  self-confident  about  learning. 

Assessment 

Assessment  is  the  process  of  gathering  information  or  observable  evidence 
of  what  the  learner  can  do.  This  section  examines  functions,  need  for 
change,  some  alternative  approaches,  and  a systems  approach  related 
to  assessment. 

Functions 

The  root  word  of  assessment  means  to  "sit  with"  a learner,  paying  close 
attention  to  the  evidence  of  a learner's  thoughtful  understanding.  "And 
thoughtful  understanding  implies  being  able  to  do  something  effective, 
transformative,  or  novel  with  a problem  or  novel  situation"  (Wiggins,  1989, 
p.  34).  The  background  from  which  a teacher  can  make  appropriate 
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assessments  is  rooted  in  that  teacher’s  deep  understanding  and  abilities  to 
communicate  that  understanding  to  students.  Eisner  (1985,  1991)  defines 
that  deep  understanding  as  connoisseurship  — the  ability  to  perceive 
subtle  qualities  and  nuances  not  available  to  most.  The  connoisseur  must 
then  use  criticism,  which  is"  ...[the]  art  of  saying  useful  things  about 
complex  and  subtle  objects  and  events  so  that  others  less  sophisticated  or 
sophisticated  in  different  ways,  can  see  and  understand  what  they  did  not 
see  and  understand  before"  (1991,  p.  3). 

Resnick  and  Resnick  (1985)  discuss  assessment  and  curriculum  as  the  two 
key  components  that  can  be  changed  to  effect  school  improvements.  The 
evidence  and  common  sense  tell  us  that  what  is  assessed  will  be  taught. 
Although  the  practice  of  teaching  to  the  test  may  be  frowned  on  in 
educational  circles,  as  Nickerson  (1989)  says,  "frowning  on  a practice  has 
seldom  been  an  effective  deterrent  if  the  incentives  to  engage  in  it  are 
strong"  (p.  3).  The  incentives  to  increase  test  scores  are  substantial, 
propelled  by  competition  for  resources  and  increased  scrutiny  and  criticism 
of  the  educational  enterprise. 

Need  for  Change 

One  key  issue  in  discussions  about  assessment  is  encapsulated  in  the 
statement:  "American  children  are  the  most  tested  in  the  world  and  the 
least  examined"  (Resnick  & Resnick,  1985,  p.  17).  Their  intention  is  not  to 
eliminate  testing,  but  rather  to  improve  tests  so  that  they  demonstrate  real 
achievement,  what  we  want  students  to  know  as  a result  of  schooling. 
Assessment  determines  what  teachers  teach,  states  Wiggins  (1989),  so 
instead  of  trying  to  fight  that  tendency,  educators  should  concentrate  on 
developing  effective  tests  that  examine  those  competencies  they  think  are 
essential. 

If  the  goals  of  education  are  broad,  then  the  tests  used  to  assess 
achievement  must  address  this  breadth  (Hargreaves,  1990;  California 
Assessment  Program,  1989).  A reliance  on  only  one  form  of  assessment, 
such  as  the  multiple  choice  test,  cannot  do  the  job  fully.  There  is  an 
extensive  body  of  expressed  concern  over  the  exclusive  use  of  multiple 
choice  testing,  most  often  highlighting  the  problems  of  teaching  to  the  test, 
ignoring  untested  curricula,  segmenting  knowledge,  "decontextualizing" 
knowledge,  and  downplaying  the  role  of  the  teacher  as  professional 
(Archbald  & Newmann,  1988;  Mumane  & Raizen,  1988;  Resnick  & 
Resnick,  1985;  Wiggins,  1989). 

Teaching  includes  the  diagnosis  of  learning  concerns  and  the  development 
and  implementation  of  activities  to  address  these.  As  part  of  the  teaching 
process,  assessment  practices  must  provide  information  to  help  with 
diagnosis  and  development  activities.  Assessments  must  contain  the 
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information,  and  teachers  must  understand  and  use  the  information  to 
make  decisions.  In  the  assessment  of  self-regulatory  and  motivational 
activities  of  learners,  Snow  (1989)  states  that  some  teachers  know  how  to 
assess  learner  outcomes  in  these  areas  informally,  and  research  faces  the 
considerable  task  of  refining  tools  for  use  by  teachers  and  others  to 
formalize  and  improve  this  measurement.  Assessment  instruments  need 
careful  design,  need  to  be  developed  by  professionals  in  education,  and 
assistance  with  interpretation  must  be  available  to  those  who  use  the 
information. 

Some  Alternative  Approaches 

Performance  assessment  alternatives  are  suggested  as  a means  to  assess 
higher  order  thinking  skills  and  to  broaden  assessment  processes.  These 
alternatives  are  encouraged  in  many  subject  areas  and  implemented 
through  such  vehicles  as  portfolios,  extended  performances  and  displays. 

One  example  of  performance  assessment  is  the  Arts  PROPEL  project,  a 
long-term  developmental  project  that  involves  the  Education^  Testing 
Service,  Pittsburgh  Public  Schools  and  the  Rockefeller  Foundation 
(Gardner,  1989;  Wolf,  1989b).  The  PROPEL  project  supports  the 
development  of  performance  assessment  instruments  to  document  artistic 
learning  from  late  elementary  to  high  school  in  music,  art  and  expressive 
writing.  Some  of  the  main  tenets  include:  teachers  having  deep 
knowledge  in  the  subject  areas;  learning  centred  around  meaningful, 
significant  projects  which  may  involve  students  for  a significant  period  of 
time;  assessment  as  crucial,  but  respectful  of  the  intelligence  being 
assessed.  The  term  PROPEL  is  an  acronym  for  the  three  competencies  to 
be  assessed:  production,  perception,  and  reflection.  Other  examples  of 
performance  assessment  in  the  arts  recommend  portfolio  assessment  for  a 
variety  of  functions  to  foster  artistic  critiques,  aesthetic  reasoning,  and  to 
recognize  the  personal  values  espoused  in  artistic  expression  (Gardner  & 
Grunbaum,  1986).  Wolf  (1989a)  recommends  portfolio  assessments  to 
promote  characteristics  such  as  increased  student  responsibility,  an 
enlarged  view  of  what  is  learned,  recognition  of  personal  progress,  and  the 
emergence  of  a developmental  point  of  view. 

Systems  Approach 

Education  is  a system  of  interactions,  rather  than  a linear  process  of  cause- 
effect  relationships.  Instead  of  controlling  for  the  influence  of  teaching  to 
the  test,  or  practice  effects,  Frederiksen  and  Collins  (1989)  encourage 
educators  to  take  advantage  of  "systemic  validity,"  the  direct  and  expected 
connection  between  instruction  and  assessment.  "A  systemically  valid  test 
is  one  that  induces  in  the  educational  system  curricular  and  instructional 
changes  that  foster  the  development  of  the  cognitive  skills  the  test  is 
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designed  to  measure"  (p.  27).  A test  would  be  systemically  valid  only  if  it 
caused  improvement  in  those  skills  it  was  designed  to  measure  after  it  was 
in  place  for  a certain  period  of  time. 

Frederiksen  and  Collins  (1989)  identify  the  directness  of  cognitive 
assessment  and  the  degree  of  subjectivity  or  judgement  required  as 
important  characteristics  for  systemically  valid  tests  (p.  28).  Direct 
assessment  means  that  the  cognitive  skill  that  is  of  interest  is  directiy 
evaluated  as  it  is  expressed  in  the  performance  of  some  extended  task.  This 
"directness"  is  the  underlying  tenet  of  performance  assessment  (Stiggins, 
1987, 1988),  authentic  assessment  (Wiggins,  1989;  Calfee  & Hiebert,  1988; 
California  Assessment  Program,  1990),  and  portfolio  usage  (California 
Assessment  Program,  1990;  Wolf,  1989a,  1989b). 

The  degree  of  subjectivity  refers  to  the  degree  to  which  judgement, 
analysis,  and  reflection  is  used  in  assigning  a score  to  a test  performance. 
Frederiksen  and  Collins  (1989)  recommend  subjective  assessment  for  the 
development  of  materials  to  assess  student  performances,  and  the 
implementation  of  collaborative  systems  for  marking  and  moderating 
student  work.  They  say: 

Although  it  is  worthwhile  to  develop  objective  tests  of  important  cognitive 
outcomes,  in  general  the  state  of  the  art  does  not  permit  objective  tests  for 
directly  measuring  higher  order  thinking  skills,  problem-solving  strategies, 
and  meta-cognitive  abilities  such  as  those  involved  in  teaching,  writing, 
and  ’doing’  mathematics  (p.  29). 

The  training  materials  for  subjective  assessment  help  to  communicate  to 
teachers  and  students  the  critical  traits  to  look  for  in  examining  their  own 
or  other's  work.  This  communication  function  is  a central  component  of 
the  testing  procedures  of  the  Assessment  of  Performance  Unit  in  Britain 
(Joint  Matriculation  Board,  1984),  and  the  California  Assessment  Program 
(CAP,  1990)  and  the  Benchmarks  program  in  Toronto  (Toronto  Board  of 
Education,  1989). 

Components  of  a systemically  valid  testing  system  would  include  a set  of 
tasks,  a specification  of  primary  traits  of  both  process  and  product  that  the 
assessors  know  are  leamable,  a library  of  exemplars,  standards,  and  a 
method  for  fostering  improvement  on  the  test,  including  such  activities  as 
student  practice  (Frederiksen  & Collins,  1989).  This  assessment  system  fits 
the  practical  descriptions  of  the  testing  programs  noted  in  Britain, 
California,  and  Toronto,  and  is  a useful  format  for  the  development  of 
alternative  assessment  techniques  in  the  EQI  project. 
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Implications  for  EQI  Calgary 


The  Calgary  EQI  Project  addresses  the  cognitive  outcomes  in  the  problem- 
solving strand  of  mathematics  and  in  the  depiction  and  composition 
objectives  of  art.  These  two  sets  of  outcomes  represent  particularly 
difficult  challenges  for  assessment  because  of  the  diverse  nature  of  the 
student  outcomes  and  the  expectation  that  the  professional  judgement  of 
educators  will  be  central  to  completing  the  assessment.  The  literature 
describes  the  need  for  changes  to  accommodate  what  we  know  about 
learning,  teaching  and  assessment,  and  provides  us  with  some  promising 
new  approaches  for  developing  new  assessment  practices. 

The  need  for  tests  that  do  a better  job  of  assessing  thinking  skills  is 
widely  recognized.  The  making  of  visual  art  and  the  solving  of  problems 
in  mathematics  both  require  complex  thinking  skills.  Strategic  flexibility, 
adaptive  control,  and  achievement  motivation  are  demonstrated  in  problem 
solving  (Snow,  1989;  National  Council  of  Teachers  of  Mathematics,  1989; 
Schoenfeld,  1988;  Mumane,  1988).  A similar  set  of  cognitive  processes  is 
suggested  in  the  literature  regarding  the  production  of  visual  art  (Glenn, 
1989;  Wolf,  1989a).  Research  reviews  (Norris,  1989;  Siegler,  1989; 
Nickerson,  1989)  suggest  positive  directions  for  improvements  to  testing 
methods  used  currently  and  for  the  development  of  new  instruments  and 
strategies  to  apply  what  is  now  being  discovered  about  cognitive 
operations. 

Promising  approaches  that  link  the  assessment  process  with  instructional 
processes  and  focus  on  the  development  of  teacher  skills  and  judgement 
are  under  way  in  many  formats  around  the  world.  Some  programs  of  this 
nature  are  formal  programs  instituted  by  governing  agencies  and  some  are 
simply  individual  teacfier  assessment  activities.  Psychometric  researchers 
(Snow,  1989;  Nickerson,  1989)  and  educators  involved  directly  in  school 
change  efforts  (Wiggins,  1989;  Stiggins,  1988;  Gardner,  1989;  National 
Council  of  Teachers  of  Mathematics,  1989)  present  a consistent  emphasis 
on  the  development  of  teacher  expertise  in  the  process  of  improving 
assessment,  both  in  external  assessment  developed  by  governing  agencies 
and  in  teacher  developed  testing. 

A ’’systems  approach”  to  assessment  such  as  described  by  Frederiksen  and 
Collins  (1989)  provides  a model  to  accommodate  the  demands  of 
assessment  in  the  production  of  visual  art  and  in  mathematical  problem 
solving.  First,  the  systems  model  adopts  a view  of  learning  as  a complex 
meaning-making  activity  that  has  many  representations,  depending  on 
the  skills,  experience  and  environment  of  the  learner.  Second,  assessment 
is  facilitated  by  a teacher  whose  informed  professional  judgement  is 
essential  for  the  assessment  of  student  cognition.  Third,  assessment  is 
focused  on  the  actual  behaviours  that  demonstrate  the  accomplishment 
of  the  learning  objective  intended,  often  the  extended  response  to  a 
complex  challenge  or  problem. 
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CHAPTER  3:  EQI-ART 


Images  are  records  of  created  meaning  ...  the  image  is  an  attempt 
of  the  mind  to  penetrate  the  meaning  of  the  world  and  the 
meaning  of  being  human  within  it.  Even  in  the  most  primitive 
attempts  at  the  symbolic  language  of  images,  there  is  evidence  of  a 
mind  at  work:  selecting,  rejecting,  arranging,  creating:  inventing 
the  clothing  for  the  idea. 

MacLeod  (1989,  p.  7) 

Introduction  to  EQI-Art 

Student  outcomes  in  response  to  the  depiction  and  composition  objectives 
of  the  Alberta  Education  art  curriculum  are  observed  in  the  art  produced 
by  students.  However,  there  are  currently  no  assessment  devices  available 
that  are  based  on  commonly  understood  expert  standards  regarding  the 
quality  of  student  artwork.  The  EQI-Art  project  was  designed  to  fill  this 
void,  by  developing  quality  indicators  that  assist  both  teachers  and 
students  in  their  respective  tasks  of  teaching  and  learning. 

The  EQI  project  has  defined  quality  indicators  as  observable  characteristics 
of  excellence  established  through  consensus  of  professional  judgement 
among  practising  teachers.  The  EQI-Art  project  provides  documentation 
of  those  observable  characteristics  in  student  art  which  should  be 
recognized  and  encouraged  and  which  assist  in  the  development  of 
teacher  and  student  expertise  in  art.  It  was  designed  as  collaborative 
research  involving  eight  expert  art  teachers  (EQI-Art  teacher  researchers), 
four  art  specialists  (EQI-Art  Advisory  Committee),  and  sixteen  practising 
art  teachers  (field  test  and  pilot  teachers).  The  list  of  the  participants  in  the 
EQI-Art  project  appears  in  Appendix  A. 

Quality  indicators  of  student  art  revealed  through  collegial  exchanges  of 
expert  professional  judgement  are  the  focus  of  the  project.  Two  primary 
considerations  underlie  the  EQI-Art  project  research  and  development:  the 
description  of  student  artwork  that  demonstrates  quality,  and  the 
development  of  teacher  expertise  in  identifying  quality  work  in  the  visual 
arts. 


Student  Art  Work 

Through  the  examination  of  student  work  we  expect  to  document 
observable  characteristics  on  which  student  performance  should  be 
properly  judged.  Curriculum  guides  and  support  materials  typically 
contain  significant  detail  regarding  content  to  be  mastered,  skills  to  be 
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practised,  activities  to  be  engaged  in,  and  resources  to  be  made  available. 
Expectations  regarding  quality  outcomes,  however,  are  practically  non- 
existent, leaving  one  with  the  possible  interpretation  that  quality  outcomes 
do  not  really  matter  much,  that  the  process  of  engagement  alone  is  most 
important.  Or,  more  positively,  the  impression  is  given  that  teachers  have 
the  ability  to  assess  quality  performance  in  art  and  do  not  need  assistance 
in  curriculum  documentation. 

Teacher  Expertise 

This  second  consideration  is  the  core  of  student  assessment  in  art,  although 
the  issues  are  somewhat  different  in  elementary  and  secondary  programs. 

In  elementary  schools  there  are  few  art  specialists,  and  the  majority  of 
elementary  teachers  are  teaching  and  assessing  student  performance  in  art 
courses  with  little  expertise  or  training  in  art.  In  the  secondary  schools, 
there  are  proportionately  more  art  teachers  with  expertise  and  training  in 
the  subject,  and  the  elective  nature  of  the  school  programs  may  encourage 
better  performance  simply  due  to  the  element  of  choice.  However,  the 
consistency  of  performance  expectations,  particularly  through  the 
collaboration  of  professional  thinking,  is  not  systematically  provided  for  in 
support  materials  available  to  these  teachers. 

The  purpose  of  the  EQI-Art  project  is  to  provide  assessment  devices  that 
describe  quality  student  performance  in  art,  and  do  so  in  a manner  that 
uses  and  develops  teacher  expertise.  The  project  is  founded  on  the 
principle  that  the  articulation  and  communication  of  quality  indicators  for 
art  production  will  lead  to  improved  teaching  behaviours  and  student 
learning.  The  following  model  depicts  these  ideas. 
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EQI-ART  Model  of  Assessment 


The  model  presents  the  art  work  as  the  central  result  of  the  practice  of 
student  skill  and  knowledge,  the  application  of  student  judgement,  and  the 
influence  of  teacher  assessment  sMs  and  EQI  models.  The  outer  ring 
suggests  the  shaping  interactions  of  teacher  practice,  reflection,  curriculum 
content,  and  the' students'  desires  to  create  significant  images. 

This  model  presents  assessment  as  a rich  and  complex  challenge  that 
involves  the  teacher,  the  student,  and  the  work. 

Table  1 on  the  following  page  outlines  the  project  personnel,  tasks  and 
results  from  1989  to  1992.  A sample  of  EQI-Art  project  materials  is 
included  in  Appendix  B. 
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Table  1:  EQI-Art  Project  Outline 
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Year  1:  Development 


In  the  first  year  of  EQI-Art,  teachers  and  advisors  examined  student  art  ^d 
identified  characteristics  of  various  levels  of  quality.  Subsequently,  several 
formats  were  developed  for  using  this  information  in  assessing  art.  There 
were  four  stages  to  die  year's  work:  1)  identifying  characteristics  of 
quality,  2)  raising  concerns  about  predetermined  characteristics, 

3)  preparing  a framework  for  art  assessment,  and  4)  developing  a 
prototype  handbook. 

Identifying  Characteristics  of  Quality 

During  the  first  project  meetings  in  1989,  EQI-Art  teacher  researchers  and 
specialists  reviewed  class  sets  of  student  art  and  clustered  the  works  into 
"below  average,"  "average,"  or  "above  average"  groupings.  In  each  cluster 
of  student  art,  the  teacher  researchers  and  specialists  then  attempted  to 
identify  quality  characteristics.  Although  there  was  a lot  of  agreement  and 
consistency  in  the  clustering,  it  was  far  more  difficult  to  identify 
"observable  characteristics"  that  indicated  the  quality  of  individual  student 
work. 

After  reviewing  many  sets  of  student  artwork,  and  categorizing  each  piece 
by  its  apparent  quality,  EQI-Art  teacher  researchers  and  specialists  outlined 
some  initial  characteristics  of  quality  that  could  be  seen  in  student  art 
works.  The  characteristics  were: 

• significant  use  of  elements  of  design  - colour,  line,  shape,  space,  value; 

• application  of  the  principles  of  design  - "presentation,"  balance, 
organization,  emphasis,  repetition,  variation,  etc.; 

• demonstration  of  technical  control  - appropriateness,  competence, 
creativity;  and 

• evidence  of  thought  - communication  of  a thought,  emotion; 
complexity  of  work,  novel  use  of  symbol  or  medium. 

For  each  of  the  characteristics,  the  teacher  researchers  and  specialists 
developed  comprehensive  lists  of  observations  that  could  be  made  about 
student  art.  Teachers  tried  out  these  lists  in  their  classrooms  and  in  project 
meetings.  Teachers  found  the  lists  helpful  in  only  a few  instances.  For 
example,  the  lists  were  helpful  where  the  assigned  art  problem  was  very 
structured  and  limitations  were  imposed  on  media,  size  and  technique.  In 
some  cases,  however,  the  identified  characteristics  listed  above  were  not 
seen  to  have  any  necessary  connection  with  "quality"  in  the  particular 
works. 
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It  also  seemed  that  a single  set  of  characteristics  was  not  sufficient  for  the 
variety  of  works  students  present.  Therefore,  a set  of  categories  of 
artmaking  — abstraction,  working  from  observation  or  from  memory,  and 
personal  expression  or  interpretation  — were  developed,  within  which 
more  specific  characteristics  could  be  observed  and  outlined.  These  new 
lists  were  tried  in  classrooms  and  again,  they  proved  to  be  of  limited  value. 

The  lists  were  then  restructured  into  the  form  of  questions  that  required  the 
user  to  respond  regarding  each  of  the  characteristics  that  could  be  seen  in 
student  art  work.  For  example,  questions  included:  "What  are  the 
methods  used  to  illustrate  depth?"  "How  is  your  attention  drawn  to  the 
centre  of  interest?"  and  "How  is  unity  achieved?"  When  reviewed  by  the 
teacher  researchers,  it  was  decided  that  this  structure  would  be  useful  only 
for  those  teachers  who  have  considerable  background  in  art,  and  many 
teachers  do  not. 

In  attempting  to  provide  sets  of  observable  characteristics  for  several 
different  types  of  artmaking,  the  group  became  frustrated  with  the 
constraints  that  were  being  established,  and  the  unwieldy  nature  of  the 
instruments.  In  addition,  student  art  often  did  not  stricdy  "fit"  the  four 
categories  that  had  been  defined. 

Raising  Concerns  about  Predetermined  Characteristics 

As  a result  of  the  concerns  and  frustrations  with  lists  and  questions, 
EQI-Art  project  committees  felt  that  the  checklists  and  sets  of  questions 
were  not  adequate  to  the  task  of  assessing  student  artwork  and  the 
following  concerns  were  raised: 

• a comprehensive  list  of  everything  one  could  see  in  art  works  was 
virtually  impossible  to  create  because  of  the  variety  in  art; 

• if  there  were  such  a list,  no  teacher  would  have  time  to  use  it; 

• a checklist  style  might  cause  teachers  to  concentrate  on  elements  in 
the  art  that,  while  easy  to  observe,  contribute  little  to  the  overall 
quality; 

• a checklist  approach  might  suggest  that  standardization  is  desirable  in 
student  artwork,  and  we  wished  to  celebrate  diversity; 

• there  is  great  variety  among  artworks  that  are  all  considered  "superior," 
or  "below  average"; 

• teachers  need  assistance  in  learning  to  observe  art,  rather  than  a 
technical  tool  that  will  have  limited  applicability; 
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• it  is  important  to  include  the  intended  meaning  of  the  student  artist  in 
the  practice  of  art  assessment; 

• to  assess  student  artwork  we  must  include  the  teacher  intentions  (or 
lesson  presentation),  and  the  classroom  constraints;  and 

• categories  of  art  that  would  better  capture  the  diversity  of  student 
artmaking  were  suggested:  observation,  interpretation,  response, 
special  communication  of  the  self,  and  design  problems. 

In  response  to  these  concerns,  three  basic  criteria  were  established  to 

provide  a framework  for  observing  quality  indicators  of  student  art: 

• RELATIONSHIPS  of  the  elements  and  principles  of  art 

• HANDLING  of  the  media  used 

• MEANING  or  artist’s  intention  in  creating  the  work 

Five  categories  of  art  making  were  defined: 

• DIRECTED  OBSERVATIONS:  such  works  include  contour  drawings, 
still-life,  drawings  from  observed  objects,  from  nature,  from  art,  and 
from  designed  objects  such  as  toys,  machines,  or  tools,  etc. 

• NARRATIVE  WORKS:  these  works  show  the  student's  response  to 
ideas  from  language  arts,  social  studies,  sciences,  etc.  Any  work  that 
tells  a story  may  be  included  in  this  category. 

• RESPONSES  TO  WORKS  OF  ART:  such  images  come  from 
experiences  of  art  works.  Teachers  will  sometimes  have  students 
study  the  work  of  a particular  artist  like  Van  Gogh,  or  a special  art 
movement  like  Impressionism,  and  then  have  students  use  such  styles 
to  make  new  images.  Included  are  art  works  that  result  from  the  study 
of  the  art  of  other  cultures. 

• SPEQAL  COMMUNICATIONS  OF  THE  SELF:  these  are  art  works 
that  reveal  student's  dreams,  fantasies,  deep  feelings  about  experiences 
and/or  social  problems,  and  memories. 

• DESIGN  PROBLEMS:  works  in  this  category  usually  result  from  the 
challenge  set  by  the  teacher  to  create  pattern,  or  to  analyze  balance,  or 
symmetry,  or  to  express  ideas  through  special  colour  selections,  i.e. 
warm  and  cool. 
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Preparing  a Framework  for  Art  Assessment 

The  EQI-Art  personnel  rejected  the  use  of  checklists  for  assessing  student 
art  work  because  this  structure  was  too  limiting  for  the  variety  of  artwork 
presented,  and  because  the  checklist  ’’style”  did  not  recognize  the 
effectiveness  of  individual  professional  judgement.  A narrative  style  of 
assessment  was  selected  — one  which  modelled  expert  professional 
judgement  based  on  connoisseurship  and  education^  criticism  (Eisner, 
1985).  The  expert  judgement  of  practising  art  teachers  was  presented  as  a 
written  critique  individually  prepared  for  each  of  the  art  works  chosen  by 
the  project.  The  critique  focused  on  the  three  basic  criteria:  relationships, 
handling,  and  meaning. 

The  quality  of  a work  of  art  is  found  in  the  completed  work  itself,  but  the 
work  comes  out  of  a set  of  conditions  that  give  it  additional  meaning  for 
the  viewer.  In  the  educational  setting,  an  understanding  of  these 
conditions  provides  teachers  with  increased  understanding  about  the 
process  of  making  and  assessing  art. 

The  identification  of  quality  indicators  in  student  art  is  practised  within  a 
school  setting  that  consists  of  conditions  — some  of  which  can  be  largely 
controlled  by  the  teacher  and  some  of  which  are  controlled  by  the  student. 
These  conditions  are  addressed  in  the  EQI-Art  project. 

In  order  to  assess  student  works  of  art,  teacher  researchers  wanted  to 
know  more  about  the  conditions  under  which  the  art  was  produced: 

• what  was  the  assignment? 

• how  much  background  do  students  have/receive  to  do  the 
assignment? 

• how  much  time  was  allowed? 

• what  materials  did  the  students  have  available? 

At  the  beginning  of  the  project,  EQI-Art  teacher  researchers  brought  in  the 
work  of  their  own  classes  and  explained  the  conditions  of  the  classroom 
assignment.  Later,  the  teacher  researchers  at  each  grade  level  developed  a 
joint  lesson  plan  which  was  implemented  and  the  results  brought  back  to 
the  committee.  This  latter  plan  was  abandoned  because  the  teachers  found 
that  lessons  often  had  to  be  modified  significantly  because  of  resources 
available  and  student  interests  at  the  time.  Quality  artwork  seemed  to 
arise  from  lessons  that  engaged  students  in  the  shaping  of  the 
background  information  and  in  the  interpretation  of  the  art  problem 
itself.  The  possible  use  of  prescriptive  lesson  plans  as  a factor  in  the 
production  of  quality  artwork  was  rejected  as  it  did  not  fit  the 
constructivist  view  of  learning  nor  the  belief  in  teacher  professional 
judgement  as  the  basis  for  educational  decisions. 
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It  was  important,  however,  to  know  about  the  lesson  that  was  presented  in 
the  classroom.  With  all  samples  of  student  artwork,  we  decided  to  provide 
a descriptive  account  from  the  teacher  that  included  an  explanation  of  the 
lesson,  the  teacher  expectations,  and  the  resources  available  (time, 
materials).  A sample  descriptive  account  from  a grade  2 lesson  follows: 


Lesson; 

The  teacher  challenged  students  to  reflect  that  environments  are  altered 
by  natural  forces  and  to  observe  that  shapes  can  suggest  movement. 
Students  were  to  observe  examples  of  the  changing  patterns  created  by 
various  objects  moving  through  water  and  to  note  how  light  is  reflected 
when  water  is  moving. 

Using  oil  pastels,  the  students  were  asked  to  depict  movement 
through  water.  Students  observed  many  examples  from  art  and 
from  the  teacher's  picture  file. 

Teacher  Expectations: 

The  teacher  expected  that  the  students  would: 

• imagine  an  animal  swimming  through  water 

• create  a two-colour  oil  pastel  drawing  showing  how  water 
changes  when  an  animal  swims  through  it 

• make  patterns  that  show  light  and  shadow  on  the  surface  of 
the  water 

• indicate  the  movement  of  the  water's  wake 
Resources: 

Materials:  Coloured  construction  paper,  12"  x 24" 

Oil  pastels 

Time:  Two  45-minute  class  periods  in  the  fall  term 

Figure  2 

Another  condition  for  apprehending  the  quality  of  student  artwork  is  the 
student  intention  in  producing  the  art. 

Teachers  who  know  their  students  well  can  speculate  and  make  quite 
accurate  inferences  regarding  student  intentions  much  of  the  time.  The 
EQI-Ait  teacher  researchers  felt  that  asking  students  to  provide  intentions 
directly  would  provide  valuable  information  for  the  teachers  in  assessing 
the  artworks  and  might  also  be  valuable  for  students  in  forming  and 
clarifying  their  ideas.  Some  students  were  asked  to  write  about  their 
intentions  and  some  were  audiotaped  and  videotaped.  The  written  form 
seemed  to  be  most  useful.  Figure  3 presents  an  example  from  the  grade  2 
lesson  described  above. 
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Figure  3 


Student  Intention: 

/ drew  a beaver  swimming  in  a pond.  Because  he  is  moving  his 
feet  the  water  is  going  away  in  a wake.  1 drew  his  leg  and  had  to 
draw  these  waves  because  he  was  moving.  I drew  some  big 
circles  far  away.  They  are  things  in  the  water  that  isn ’t  moving. 

Teacher  Observation: 

The  beaver  shape  is  almost  centred  while  the  page  is  divided  into 
three  horizontal  divisions  giving  the  work  a feeling  of  order  and 
balance.  The  colour  reversals  in  the  large  shapes  — dark  red 
centres,  white  centres,  some  with  borders,  some  without  — are 
unusual  and  give  variety  and  rhythm  to  these  patterns. 

The  series  of  small  white  shapes  arranged  in  orderly,  double- 
diagonal rows  on  either  side  of  the  beaver  sharpen  the  contrast 
and  create  a sense  of  movement  by  leading  the  eye  to  the  central 
beaver  shape. 

The  blue  outlines  and  direction  lines  and  the  single-colour 
background  make  unity  in  the  work, 

Original  is  12" x 18" 

Figure  4 
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Developing  a Prototype  Handbook 

Toward  the  end  of  the  first  year,  the  components  of  ’’quality  indicators  in 
student  work”  and  "conditions  of  quality  art  production”  were  put 
together  in  the  form  of  a prototype  ’’Qudity  Indicators  Handbook.”  The 
handbook  reflects  Eisner’s  model  of  connoisseurship  and  educational 
criticism  (Eisner,  1985).  The  intention  of  the  handbook  is  to  illuminate 
the  process  of  assessment  through  modelling  expert  art  assessment 
practice.  In  order  to  provide  this  modelling,  the  handbook  prototype 
contained  teacher  expectations,  classroom  conditions,  and  included  colour 
reproductions  of  selected  student  work  that  expert  art  teachers  would 
consider  ’’superior,”  ’’above  average,”  ’’average”  and  ’’below  average.” 
Each  artwork  was  accompanied  by  student  intentions  and  expert  analysis 
focusing  on  relationships,  handling  and  meaning. 

The  EQI-Art  teacher  researchers  used  and  critiqued  the  prototype  model, 
making  a number  of  suggestions  regarding  improvement.  Three  very 
important  suggestions  were  implemented. 

First,  it  was  recommended  that  we  remove  the  qualifications  ’’superior  ...  to 
below  average,”  and  present  only  those  works  that  were  of  sufficient 
quality  to  allow  extended  critique  regarding  the  quality  present.  Several 
factors  were  behind  this  recommendation.  There  was  a concern  that  the 
presentation  in  published  form  of  materials  that  would  be  held  up  as  ’’poor 
examples”  would  detract  from  the  emphasis  on  quality.  A difficulty  in 
copyright  clearance  in  presenting  student  work  in  a negative  light  was 
also  anticipated.  Furthermore,  the  experience  of  the  researchers  suggested 
that  by  contemplating  the  organization  of  successful  works,  teachers 
would  gradually  build  up  a positive  repertoire  of  quality  indicators  they 
could  readily  notice. 

Second,  the  teacher  researchers  recommended  that  some  kind  of  summary 
list  appear  in  the  document  to  provide  teachers  with  a vocabulary  for 
addressing  the  characteristics  of  ’’Relationships,”  ’’Handling,”  and 
’’Meaning.” 

Third,  there  was  considerable  discussion  regarding  the  value  of  providing 
handbooks  for  the  specifically  identified  grades  8 and  11.  At  the  junior 
and  senior  high  school  levels,  there  are  often  several  grade  levels  in  the 
same  classroom,  and  in  most  schools  only  one  teacher  offers  all  of  the  art 
courses.  In  addition,  the  curriculum  objectives  within  the  junior  high  and 
senior  high  are  quite  similar.  It  was  recommended,  then,  that  there  be 
’’junior  high”  and  ’’senior  high”  handbooks,  which  could  more  widely  used 
by  art  teachers. 
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These  recommendations  were  implemented  in  the  development  of  Quality 
Indicators  Handbooks  for  grades  2 and  5,  junior  high,  and  senior  high 
school  grades.  Each  handbook  contained  several  teacher  explanations  of 
art  lessons  related  to  the  curriculum  at  that  grade  level.  With  each  lesson 
explanation  is  a set  of  colour  reproductions  of  quality  student  art.  Each 
reproduction  is  accompanied  by  "student  intentions"  and  "expert 
observations"  of  the  quality  indicators  that  can  be  seen  in  the  artwork. 

The  format  of  the  handbooks  invites  the  users  to  become  involved  in  the 
process  of  ''seeing  with  expert  eyes"  and  to  model  the  same  process  in  their 
own  assessment  practice. 

Year  2:  Field  Testing 

In  the  second  year  of  EQI-Ait,  teachers  and  advisors  developed  and 
carried  out  a plan  for  the  field  testing  of  the  EQI-Art  handbooks  in  local 
classrooms.  There  were  four  stages  to  this  year’s  work:  1)  field  testing  the 
handbooks,  2)  responses  from  the  field-test  teachers,  and  3)  revising 
the  handbooks  prior  to  piloting. 

Field  Test  of  the  Quality  Indicators  Handbooks 

Eight  successful,  professionally  active  teachers  from  the  two  Calgary 
boards  were  selected  to  field-test  the  Quality  Indicators  Handbooks  in 
their  classrooms  during  February  and  March  1991.  These  teachers  were 
given  a brief  inservice  and  invited  to  use  the  handbooks  in  many  different 
ways  in  their  art  teaching  and  assessment. 

The  EQI-Art  field  test  was  designed  to  encourage  practitioners  to  use  and 
examine  the  Quality  Indicator  Handbooks  within  the  practice  of  their 
teaching  situations  in  order  to  validate  the  research  of  the  committees.  The 
handbooks  are  intended  to  be  used  as  a resource  for  the  guidance  of  the 
practice  of  assessment  in  the  learning  environment. 

The  stated  objectives  of  the  field  test  were  to: 

• identify  the  strengths  and  weaknesses  of  the  EQI-Art  Quality 
Indicators  Handbook  (QIH)  as  a teacher  resource; 

• collect  information  from  teachers  who  would  be  defined  as  "specialist" 
and  "generalist"  with  regard  to  response  to  and  use  of  QIH  materials; 
and 

• collect  data  about  classroom  conditions  that  contribute  to  the 
development  of  quality  student  art. 
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Field  test  teachers  were  involved  in  the  following  tasks: 

• Field  test  teachers  prepared  a written  assessment  on  a work  of  student 
art  prior  to  the  use  of  the  QIH,  then  prepared  a second  assessment  on 
another  work  at  the  end  of  field  testing.  These  two  assessments  were 
analyzed  by  the  EQI-Ait  advisors. 

• The  selected  teachers  field  tested  the  initial  version  of  a survey, 
"Teacher  Survey  of  Assessment  Attitudes  and  Skills,"  designed  to 
answer  questions  about  relationships  between  the  characteristics  of 
experience,  confidence,  and  expertise  in  art  assessment,  and  the 
belief  systems  of  art  teachers.  Results  are  found  in  Appendix  B. 

• Field  test  teachers  responded  to  the  QIH  materials  by  writing  directly 
in  the  handbooks  to  suggest  changes,  additions,  deletions  to  any  parts, 
and  by  documenting  classroom  practice  using  the  QIH. 

• If  teachers  used  the  QIH  materials  in  evaluative  procedures,  they  were 
asked  to  share  the  manner  or  structures  developed  for  this  purpose. 

• If  teachers  involved  students  in  using  the  QIH  materials  — for  example 
in  self-assessment  or  peer  assessment  — they  were  asked  to  document 
the  use  in  written  accounts,  audiotape  or  videotape. 

Responses  to  the  Field  Test 

At  the  end  of  the  field  test,  information  was  collected  on  teacher  attitudes 
and  skills,  uses  made  of  the  handbooks,  and  concerns  teachers  had  in 
implementing  the  assessment  practices.  Responses  from  the  field  test 
teachers  included  perceived  effects  of  the  use  of  the  QIH  in  the  classes, 
recommended  revisions  to  the  handbooks,  and  suggestions  for  inservicing 
pilot  teachers  for  next  year. 

A few  of  the  field  test  teacher  comments  related  to  the  use  of  the 
handbooks  follow: 

• I found  that  when  they  [students]  were  doing  their  artwork  — they 
were  standing  back  and  looking  at  how  they  could  make  changes 
and  thinking  about  what  they  could  do  to  improve  their  work. 

• I referred  to  the  pictures  in  the  handbook  and  we  discussed  them  in 
terms  of  quality  indicators,  and  they  [students]  would  do  whatever  it 
was  that  we  were  doing  ...  but  they  would  really  tune  into  the  things 
that  we  had  been  discussing. 
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• I thought  the  intention  [question]  was  excellent  for  the  student ...  I 
think  it  made  the  kids  focus  in,  at  least  in  junior  high,  on  the 
objectives  [of  the  lesson]  and  they  were  conscious  of  this 
throughout. 

• I think  probably  technically  they  [students]  would  look  for  things 
more  easily.  They  would  really  start  to  look  at  compositions  and 
they  would  realize  ...the  influences  they've  had  from  other  artists  ... 
but  maybe  that's  just  because  I'm  prompting  them  more  too.  But  I 
would  say  'yes'  it's  [EQI-Art]  making  a difference. 

The  field  test  teachers  were  also  involved  in  the  process  of  looking  at 
changes  in  their  own  assessment  practice.  Prior  to  beginning  to  work  with 
the  handbooks,  each  teacher  was  asked  to  write  a critique  of  a student  art 
work.  At  the  end  of  the  field  test,  teachers  were  again  asked  to  write  a 
critique,  considering  a different  student  piece.  A comparison  indicated  an 
increase  in  specificity.  In  the  earlier  example,  a teacher  might  write  ”...  a 
good  use  of  colour,"  but  in  the  second  example,  write  "...  blue  adds 
definition  to  the  figure  and  also  gives  the  snowman  a 'cool'  feeling." 

Comments  and  recommendations  provided  by  the  field  test  teachers  were 
used  to  clarify  some  of  the  instructions  in  the  handbook,  to  add  some 
insights  to  the  art  critiques  included,  and  to  reformat  some  of  the  materials 
for  easier  classroom  use.  The  field  test  response  of  teachers  and  students 
to  the  EQI  Quality  Indicators  Handbooks  was  enthusiastic.  Improved 
practice  was  noted  by  the  field  test  teachers  themselves  in  a majority  of 
cases. 

Revisions  to  the  Quality  Indicators  Handbooks  Prior  to  Piloting 

The  initial  handbooks  used  for  field  testing  included  a limited  number  of 
samples  of  student  art,  representing  only  two  or  three  of  the  five  art  work 
categories  identified  by  the  EQI-Art  project.  Given  the  positive  response 
to  the  style  and  contents  of  the  handbooks,  the  project  team  decided  to 
complete  the  presentation  of  all  categories  for  the  pilot  year. 

To  obtain  the  art  works  necessary  for  this  expansion,  an  invitation  was 
sent  to  all  art  teachers  in  the  two  school  districts  to  submit  sets  of  student 
work  for  consideration.  A significant  collection  of  work  was  submitted  for 
consideration.  The  EQI-Art  teacher  researchers  and  specialists  who 
prepared  the  first  edition  of  the  handbooks  prepared  the  pilot  editions 
during  the  month  of  June  1991. 

Initial  handbooks  contained  only  one  grade  level,  or,  as  in  junior  and 
senior  high,  only  one  set  of  grades.  Field  test  teachers  requested  that  all 
grade  levels  be  included  in  one  handbook  so  that  they  could  be  used  to 

EQI  - Art  & Mathematics  30  Calgary  Board  of  Education 

Calgary  Catholic  Board  of  Education 


illustrate  a development  of  art  practice  and  performance  throughout  the 
grades.  The  field  test  teachers  suggested  that  the  opportunity  to  look  at 
the  collection  of  work  from  all  grades  would  benefit  students  as  well  as 
classroom  teachers. 

Consultants  recommended  that  the  project  leadership  reconsider  the 
decision  not  to  represent  a variety  of  qualities,  from  below  average  to  the 
truly  outstanding.  The  consultants  suggested  that  the  handbooks  would 
be  more  effective  if  they  help  generalist  teachers  to  "recognize  poorer 
performance  along  the  various  dimensions"  (Maguire  & Rogers,  1991).  The 
expert  teachers'  experience,  however,  indicated  that  a repertoire  of  good 
examples  of  student  work  encouraged  improved  performance.  Teachers 
and  students  in  the  pilot  group  appeared  to  learn  more  from  the 
consideration  of  successful  works  rather  than  from  the  analysis  of  inferior 
or  incomplete  works.  In  addition,  copyright  laws  do  not  permit  the 
presentation  of  individual  student  work  in  a negative  light.  Consequently, 
the  project  leadership  decided  to  stay  with  the  original  decision,  to  present 
only  works  which  were  of  good  quality  for  each  grade  level.  Negative 
examples  of  student  art  work  were  not  included. 

However,  at  the  beginning  of  every  handbook,  there  would  be  a listing  of 
the  most  obvious  quality  indicators  — relationships,  handling  and  meaning 
— with  characteristics  that  can  be  observed.  These  outlined 
characteristics  should  assist  a non-expert  art  teacher  in  identifying  the 
dimensions  of  various  qualities  as  well  as  in  developing  a vocabulary  to 
use  in  providing  art  criticism. 

Year  3:  Pilot  Testing 

At  the  end  of  the  second  year,  the  EQI-Art  project  advisors  revised  the 
initial  handbooks  by  increasing  the  number  of  art  samples,  and  by 
including  all  grade  levels  in  one  volume.  In  order  to  increase  the 
awareness  of  student  thinking  and  to  capture  teacher  reflections  on  art 
assessment,  the  art  advisors  incorporated  reflective  journals  for  both 
teachers  and  students  into  the  pilot  plans. 

In  the  third  year  of  EQI-Art,  teachers  pilot  tested  the  handbooks  in  local 
classrooms.  A variety  of  activities  helped  gather  information  about  the 
value  and  usefulness  of  the  handbooks.  These  two  sections  will  describe 
the  activities:  1)  pilot  testing  the  handbooks,  and  2)  information 
gathering  and  analysis. 

Pilot  Testing  the  Handbooks 

From  September  1991  until  the  end  of  January  1992,  eight  practicing  art 
teachers  from  CBE  and  the  CCBE  participated  as  pilot  teachers.  As  in  the 

EQI  - Art  & Mathematics  31  Calgary  Board  of  Education 

Calgary  Catholic  Board  of  Education 


field  test,  the  selected  teachers  were  successful  and  professionally  active, 
and  participation  was  on  a voluntary  basis. 

The  objectives  of  pilot  testing  were  to: 

• identify  changes  in  the  assessment  skills  and  attitudes  of  teachers 
using  the  EQI-Ait  Quality  Indicators  Handbook  (QIH); 

• identify  changes  in  the  assessment  skills  and  attitudes  of  students 
using  the  EQI-Art  QIH; 

• collect  information  from  teachers  who  would  be  described  as 
"specialist"  (teachers  with  a university  degree  in  the  field  of  art  or  art 
education)  and  "non- specialist"  regarding  their  response  to  and  use  of 
QIH  materials; 

• collect  information  about  pilot  classroom  conditions  that  support  or 
hinder  the  development  of  quality  student  art; 

• provide  an  indication  of  the  changes  in  the  quality  of  student  art  work 
that  may  be  expected  as  a result  of  teacher  and/or  student  use  of  the 
QIH;  and 

• identify  the  strengths  and  weaknesses  of  the  EQI-Art  QIH  as  a teacher 
resource. 

Pilot  teachers  were  familiarized  with  the  background  of  the  EQI-Art 
project  and  the  development  of  the  handbook  during  a day-long  inservice 
session.  Pilot  teachers  were  asked  to: 

• use  the  handbook  as  a model  to  practice  one  or  more  assessment 
activities  in  their  classes; 

• have  students  use  the  handbook  in  classroom  activities; 

• use  the  model  of  the  handbook  to  make  one  or  more  collections  of 
their  own,  based  on  a classroom  activity  completed  during  the  pilot 
period;  and 

• critique  the  handbook  by  writing  in  the  book  itself,  by  commenting  in 
the  reflective  journals,  and  by  reporting  concerns  to  the  project 
advisors. 

Pilot  teachers  were  provided  with  instruction  and  guidance  about  writing 
journal  entries.  They  and  their  students  were  asked  to  maintain  a journal 
of  their  reflections  regarding  the  use  of  the  handbook.  The  organizing 
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questions  for  the  journals  were:  "What  did  you  learn  from  your  use  of  the 
handbook  today?"  and  "What  questions  occurred  to  you  in  your  use  of 
the  EQI  materials?" 

During  the  pilot  period,  the  art  advisors  visited  all  pilot  sites  to  see  how  the 
work  was  going  and  to  review  the  teacher  and  student  journals. 

Information  Gathering  and  Analysis 

Information  gathering  occurred  prior  to,  during,  and  following  the  piloting 
experience.  The  information  and  analysis  is  reported  here  in  four  sections: 

a.  Identifying  change  in  teacher  attitudes  and  practice 

b.  Examining  the  use  of  teacher  and  student  journals 

c.  Parent  responses  to  the  EQI- Art  pilot  and  handbook 

d.  Pilot  teacher  "Conclusions"  and  "Recommendations" 

a.  Identifying  change  in  teacher  attitudes  and  practice 

Pilot  teachers  completed  a survey  of  attitudes  and  practice  on  two 
occasions  — at  the  beginning  of  the  pilot  period,  and  at  its  conclusion. 
This  survey  was  developed  to  identify  changes  in  beliefs,  attitudes, 
and  practices  relating  to  the  use  of  QIH.  The  EQI- Art  Pilot  Teacher 
Survey  is  found  in  Appendix  B.  The  survey  provided  demographic 
data  about  experience,  background,  and  teaching  assignment,  as  well 
as  information  regarding  the  beliefs  and  practices  of  teachers 
regarding  art  teaching,  assessment,  and  student  artmaking.  The  survey 
questions  were  also  designed  to  collect  information  regarding  Eisner’s 
(1985)  ideas  of  connoisseurship  and  criticism:  "expertise," 

"confidence,"  and  "use  of  subjective  methodologies  in  art  assessment." 

Analysis:  Demographic  and  descriptive  data  from  the  survey  are 

reported  in  Appendix  B.  Analysis  by  SPSSX  yields  only 
a few  statistically  significant  changes  in  teacher  attitudes 
and  practice  between  the  two  survey  administrations, 
and  these  changes  are  not  consistent  in  the  direction  of 
more  or  less  confidence  or  expertise.  The  findings  are 
not  reported  here  because  we  feel  the  results  are  of 
limited  use,  given  the  small  number  of  participants  (11), 
the  limited  time  (4  months)  and  the  lack  of  reliability 
testing  of  the  survey. 

Pilot  teachers  also  provided  feedback  to  the  art  advisors  both 
verbally  and  in  written  form  at  the  final  pilot  meeting.  Almost  all  pilot 
teachers  commented  that  there  were  changes  in  their  practice  as  a 
result  of  EQI.  These  changes  included  plans  for  using  journal  writing 

EQI  - Art  & Mathematics  33  Calgary  Board  of  Education 

Calgary  Catholic  Board  of  Education 


in  future  classes,  awareness  of  changed  viewpoints,  needing  to  listen 
more  to  student  intentions,  and  learning  new  ways  to  talk  to  parents 
about  student  progress  in  art.  Some  of  their  written  comments  follow. 

• [I]  discovered  how  much  students  enjoy  viewing  other  students' 
work.  In  fact,  that  is  when  it  is  easiest  to  gain  and  maintain 
class  attention.  They  loved  hearing  the  artist  intentions. 

• 1 credit  art  with  being  a higher  level  cognitive  activity  than 
before.  As  children  wrote  about  their  work  they  identified  what 
must  have  been  going  through  their  minds  subconsciously  and 
brought  it  to  a conscious  level. 

• I found  that  because  of  my  involvement  with  EQl  my  way  of 
teaching  has  changed  — it  has  also  carried  over  to  other 
subject  areas  as  I have  discussed  and  worked  with  my  class  on 
ways  to  produce  Equality"  work.  They  and  I both  look  at  what 
they  are  doing  differently  — what  makes  this  quality  work? 
what  are  they  happy  with,  and  why?  what  might  they  like  to 
work  more  on?  and  what  was  difficult  for  them? 

b.  Examining  the  use  of  teacher  and  student  reflective  journals 

Pilot  teachers  were  provided  with  journals  for  their  own  use  and  for 
the  use  of  their  students  at  the  start  of  the  pilot  period.  During  the 
course  of  the  pilot,  art  specialists  from  the  two  school  systems  visited 
all  of  the  pilot  classrooms,  taking  time  to  review  teacher  and  student 
journals  and  to  write  in  them  when  appropriate.  At  the  completion  of 
the  pilot  period,  teachers  brought  these  journals  back  to  the  project 
advisors  for  analysis.  Most  of  the  pilot  teachers  commented 
positively  on  the  use  of  journals,  both  for  themselves  and  for  their 
students.  Most  teachers  identified  a connection  between  student 
writing  and  thought  processes  that  enhanced  the  quality  of  student 
work. 

Analysis:  Two  experienced  art  teachers  currently  enrolled  in 

graduate  art  education  programs  at  the  University  of 
Calgary  were  engaged  to  carry  out  an  analysis  of  the 
journals  used  in  the  EQI-Art  pilot.  These  teachers  had 
experience  in  journal  analysis,  and  had  some  familiarity 
with  the  EQI  project.  Their  conclusions  follow.  The 
complete  report  is  found  in  Appendix  B. 

In  conclusion,  we  saw  the  journals  as  holding  the  possibility  for 
being  an  excellent  means  to  foster  artistic  learning.  The  opportunity 
for  teachers  to  reflect  on  their  own  practices  was  described  by  many 
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as  being  "invaluable."  Their  constant  questioning  and  striving  to 
understand  the  project  was  made  clearer  through  their  reflections, 
observations  and  feelings  about  their  art  classes  and  students.  We 
saw  the  place  of  the  teacher  as  pivotal  in  regards  to  facilitating  the 
use  of  the  journals  in  an  approach  that  promoted  a continuing  and 
evolving  discourse.  The  students'  journals  fostered  reflection  on 
their  art  learning,  and  developed  a heightened  sensitivity  to  elements 
within  a whole.  It  is  hoped,  therefore,  that  this  encouraged  quality 
improvement  in  student  art  work. 

c.  Parent  responses  to  the  EQI-Art  pilot  and  handbook 

Parents  with  students  in  the  pilot  classes  were  contacted  and  invited 
to  discuss  the  project  experienced  by  their  children.  Meetings  were 
held  in  the  pilot  schools  with  small  parent  groups. 

Analysis:  Written  and  oral  responses  from  parents  were  positive 

towards  the  project  as  they  understood  it,  and  they 
expressed  appreciation  for  being  invited  to  respond. 
Sample  comments  follow. 

• I think  the  art  handbook  is  a great  idea.  It  helps  to  give  the 
children  an  idea  of  what  is  expected  on  certain  projects.  It 
also  helps  to  set  out  guidelines  so  the  teacher  knows  what  to 
look  for.  I think  it  will  help  the  children  to  take  pride  in  their 
art  work  and  put  more  time  and  effort  into  their  projects. 

• Good  concept.  Shows  children  what  is  expected.  The  written 
paragraph  gives  a child  a chance  to  express  their  idea  or 
intention. 

d.  Pilot  teacher  conclusions  and  recommendations 

During  the  final  meeting  of  the  pilot  teachers,  we  asked  the  teachers 
to  respond  in  writing  to  the  headings  listed  above.  We  collected 
these  responses  and  recorded  discussion  regarding  the  pilot 
experience. 

The  conclusions  reached  by  the  pilot  teachers  often  referred  to  the 
value  of  "talk,"  "language"  and  "vocabulary"  as  important  aspects  of 
focusing  student  attention  on  the  art  problem,  getting  students  to 
articulate  their  knowledge  about  art,  and  sharing  understandings  of 
quality  in  art.  Another  main  theme  was  the  opportunity  to  see  and 
hear  what  colleagues  had  to  say  about  art  assessment.  Many 
teachers  wanted  an  extension  of  the  pilot  meetings,  and  expanded 
versions  of  the  EQI  handbook. 
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• / think  the  most  positive  aspect  of  the  project  was  having  the 
students  write  down  their  thoughts  and  intentions  about  their 
work.  It  focused  their  ideas,  concerns  and  observations  about 
their  work. 

• [The  EQI-Art  Handbook]  provides  concrete  examples  of  how  to 
talk  about  specific  pieces  of  art.  Teachers  may  be  familiar  with 
terms  such  as  pattern,  unity,  etc.,  but  not  know  how  to  use  these 
in  discussion.  By  simply  reading  what  other  teachers  have 
written  about  childrens  art  makes  it  easier  to  incorporate  these 
ideas  into  classroom  practice. 

The  teachers  had  many  recommendations  for  future  developments,  for 
inservice,  for  other  ways  to  use  the  book  in  classes,  and  for  using 
journals  as  an  integral  part  of  assessment.  Some  comments  follow. 

• Inservice  of  some  form  is  essential  to  use  the  book  as  a resource. 

• It  is  really  important  that  teachers  and  students  have  the  time 
and  opportunity  to  reflect  on  their  work.  The  use  of  journals  is 
an  important  aspect  of  this.  It  is  just  as  important  that  teachers 
and  students  be  given  the  opportunity  to  dialogue  with  each 
other  and  their  peers  — EQI  certainly  heightens  one's 
awareness  of  the  possibilities  of  art  education  and  how  to  more 
accurately  and  fairly  assess  student  work. 

Follow-up  activities  suggested  for  teachers,  for  schools,  for  school  system 
art  speci^ists,  and  for  Alberta  Education  included:  that  handbooks  be 
made  readily  available,  that  time  and  support  be  given  to  teacher  meetings 
about  assessment,  and  that  the  EQI  project  people  have  a big  celebration 
to  commemorate  our  work  together. 

Pilot  teachers  were  asked  to  critique  the  handbook  and  to  bring  editorial 
and  structural  suggestions  to  the  final  pilot  meeting.  A thorough  review  of 
the  entire  handbook  was  held  during  the  meeting  and  all  comments  and 
suggestions  were  considered.  The  EQI-Art  advisors  revised  and  edited  the 
entire  handbook  during  February  and  March  1992.  The  desired  changes 
included:  an  expanded  introduction  to  the  handbook;  suggestions  for  the 
use  of  the  handbook  in  classrooms;  suggestions  for  provoking  discussion 
regarding  quality  in  art;  and  practices  that  stimulate  journal  use.  Revised 
handbooks  were  provided  to  all  teachers  and  advisors  who  worked  on  the 
project,  to  teacher  resource  departments  in  the  two  school  boards,  to 
university  personnel  for  use  in  teacher  preparation,  and  to  Alberta 
Education. 
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Findings  and  Conclusions  (Outcomes) 

The  EQI-Art  researchers,  both  teachers  and  specialists,  understood  from  the 
outset  that  the  project  required  a lengthy  period  of  development  and  trial 
and  error.  This  indeed  proved  to  be  true,  as  we  tried  and  discarded  many 
different  approaches  during  the  initial  year  and  a half.  Our  committee 
process  was  to  question,  to  research,  to  intuit,  to  test,  and  to  revise  our 
materials  and  our  ideas  about  assessing  quality  art.  The  final  handbook 
represents  the  wisdom  we  collectively  shared  regarding  the  assessment  of 
quality  in  student  art. 

The  art  advisors  considered  information  collected  from  all  sources  and  in 
many  forms  in  preparing  the  following  conclusions  about  the  EQI-Art 
project: 

• Quality  assessment  practice  supports  improved  art  making. 

• Written  and/or  spoken  student  responses,  reflections  and  thoughts 
illuminate  the  art  work. 

• Quality  art  assessment  is  a shared  experience  between  teachers  and 
students. 

• Common  vocabulary  shared  by  teachers  and  students  affects 
assessment. 

• Collection  of  quality  art  exemplars  emphasizes  diversity  as  an  attribute 
of  a quality  art  program. 

• Informed  professional  judgement  benefits  from  collegial  work. 

• Assessment  takes  time:  time  for  students  to  prepare  a quality  outcome 
and  for  teachers  to  thoughtfully  reflect  and  respond. 

• EQI  handbook  exemplars  provide  a way  to  share  expertise  and 
vocabulary,  and  to  develop  connoisseurship. 
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CHAPTER  4:  EQMVIATH 
Introduction  to  EQI-Math 

The  most  important  goal  of  mathematics  instruction  in  Alberta  is  the 
development  of  students’  abilities  to  solve  problems.  The  General  Learner 
Expectation  for  mathematics  is  that:  students  are  confident  and 
increasingly  competent  in  solving  problems  as  they  develop  and 
demonstrate  understanding  of  mathematical  concepts,  relationships  and 
procedures  (Alberta  Education,  1992,  p.  3). 

Problem-solving  abilities  and  attitudes  are  also  at  the  core  of  descriptions 
of  numeracy,  or  mathematical  literacy.  The  lack  of  mathematical  literacy 
has  been  identified  as  an  economic  liability  for  a nation,  a personal 
limitation,  and  a source  of  systematic  discrimination  for  women  and 
minorities  (National  Research  Council,  1989;  National  Council  of  Teachers 
of  Mathematics,  1989). 

One  of  the  prime  documents  recommending  change.  Curriculum  and 
Evaluation  Standards  for  School  Mathematics  (National  Council  of 
Teachers  of  Mathematics,  1989),  describes  standards  for  mathematics 
literacy  in  the  1990s  and  provides  guidelines  for  the  revision  of  curricula 
and  assessment  toward  this  literacy.  There  are  five  goals  for  students: 

1)  learning  to  value  mathematics 

2)  becoming  confident  in  one’s  own  ability  to  do  mathematics 

3)  becoming  a mathematical  problem  solver 

4)  learning  to  communicate  mathematically 

5)  learning  to  reason  mathematically 

These  considerations  clearly  extend  the  expectation  of  mathematics 
instruction  from  the  acquisition  of  specific  skills  and  processes  through  to 
positive  attitudes  toward  and  the  use  of  mathematical  reasoning  in  many 
different  life  situations.  This  expansion  of  expectations  regarding  problem- 
solving skills  and  dispositions  to  problem  solving  has  been  accompanied 
by  some  change  in  teaching  and  testing  methods  in  schools,  and  intense 
interest  in  developing  new  methods  in  response  to  the  need  for  change. 

The  EQI-Math  Project  involved  practicing  mathematics  teachers  and 
mathematics  subject  specialists  in  a search  for  new  methods  to  assess 
student  achievements  in  problem  solving.  Table  2 on  the  following  page 
outlines  the  activities  of  the  EQI-Math  project  from  1989  to  1992. 
Appendix  C contains  a sample  of  the  project  materials. 
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Table  2:  EQI-Math  Project  Outline 
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Year  1:  Development 


Quality  indicators  are  observable  characteristics  of  excellence  established 
through  consensus  of  professional  judgement  among  practising  teachers. 
Problem  solving  is  the  application  of  known  mathematical  knowledge  to 
an  unfamiliar  situation  so  as  to  arrive  at  a solution  (EQI  Project  definition). 

The  expectations  and  objectives  of  mathematics  programs  direct  attention 
to  students  applying  math  skills  and  processes  and  approaching  the  use  of 
mathematics  with  self-confidence,  a positive  attitude  about  the  efficacy  of 
math  in  understanding  real  life  situations,  and  enthusiasm.  The  indicators 
of  problem  solving  therefore  apply  to  these  aspects  of  student  outcomes  as 
teachers  observe  them  in  classrooms. 

During  the  first  year  of  the  EQI-Math  project,  teachers  and  advisors 
examined  many  aspects  of  mathematical  problem  solving,  developed  sample 
problems  and  prepared  materials  for  use  in  recording  observations  about 
student  problem  solving.  The  four  main  areas  of  activity  for  Year  1 are 
presented  in  the  following  sections:  1)  identifying  student  dispositions; 

2)  specifying  problems  for  use  with  students;  3)  developing  holistic 
marking;  and  4)  documenting  student  understanding. 

Identifying  Student  Dispositions 

We  began  our  quest  for  quality  indicators  of  mathematical  problem  solving 
with  a team  of  14  successful,  professionally  involved  mathematics  teachers 
(EQI-Math  teacher  researchers)  and  5 mathematics  subject  area  specialists 
(EQI-Math  Advisory  Committee)  from  the  Calgary  Board  of  Education 
and  the  Calgary  Catholic  Board  of  Education.  Personnel  involved  are 
listed  in  Appendix  A. 

The  selected  teachers  and  advisors  began  their  work  by  exploring  the 
concept  of  quality  in  mathematical  problem  solving.  It  was  apparent  that 
experienced  mathematics  teachers  identify  successful  problem  solvers  by 
their  dispositions  or  personal  characteristics,  in  addition  to  their  possession 
of  particular  math  skills  and  success  in  solving  math  problems.  Initial 
meetings  of  the  EQI-Math  committees  centred  around  discussions  of  what 
defined  good  problem  solving,  and  how  to  identify  good  student  problem 
solvers.  The  identification  of  good  problem  solvers  led  to  the  development 
of  a list  of  characteristics  or  traits  that  described  student  "dispositions” 

(The  National  Council  for  Teachers  of  Mathematics): 

Disposition  refers  not  simply  to  attitudes  but  to  a tendency  to 
think  and  act  in  positive  ways.  Students’  mathematical 
dispositions  are  manifested  in  the  ways  they  approach  tasks  — 
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whether  with  confidence,  willingness  to  explore  alternatives, 
perseverance,  and  interest  — and  in  their  tendency  to  reflect  on 
their  own  thinking  (NCTM,  1989,  p.  233). 

Initially,  teacher  researchers  recorded  observations  of  strong  and  weak 
problem  solvers  in  their  classes  to  obtain  descriptive  comments  about 
problem-solving  dispositions.  These  records  were  in  the  form  of  informal 
"anecdotal  records."  Teachers  found  this  task  to  be  difficult  because  there 
was  so  much  that  could  be  observed,  and  decisions  about  what  to  include 
or  not  include  in  the  anecdotes  were  complex.  The  teacher  researchers 
shared  their  experiences  with  one  another  and  discussed  anecdotal  records 
for  the  students  in  their  classrooms.  As  a result  of  these  activities,  ten 
dispositions  were  identified,  and  descriptions  of  each  disposition  were 
developed  from  teacher  observations. 

The  initial  lists  of  dispositions  prepared  by  teacher  researchers  at  both 
elementary  and  secondary  grades  were  very  similar,  so  it  was  decided  to 
proceed  with  one  list  of  dispositions  for  students  at  all  grade  levels.  The 
teacher  researchers  agreed  that  although  not  all  of  the  dispositions  would 
be  displayed  by  any  one  individual  student,  a student  who  was  a good 
problem  solver  would  display  at  least  several  of  them.  Appendix  C 
contains  the  original  ten  problem-solving  dispositions  and  descriptions. 
Seven  of  the  ten  dispositions  described  are  similar  to  those  outlined  in 
Curriculum  and  Evaluation  Standards  for  School  Mathematics  (NCTM, 
1989,  p.  233). 

Specifying  Problems  for  Use  with  Students 

The  dispositions  towards  problem  solving  are  developed  over  time  as 
students  experience  classroom  mathematics  activities.  An  important  part  of 
activities  to  identify  quality  indicators  is  the  context  for  students  to  display 
their  dispositions  and  their  problem-solving  skills.  Student  problem- 
solving dispositions  were  observed  in  the  classroom  context  where 
students  were  engaged  individually  and  in  small  groups  with  mathematics 
problems. 

Teacher  researchers  began  by  selecting  student  mathematics  problems  to 
conform  to  the  curricular  demands  of  each  grade  studied  (grades  2,  5,  8, 
and  10)  at  the  time  of  the  school  year  the  problems  were  to  be 
administered.  Problems  were  identified  for  each  strand  of  the  curriculum 
and  for  non-routine,  challenging  situations.  Teachers  used  personal 
knowledge,  resources  available  in  the  two  school  systems,  and  published 
sources  for  problem  ideas. 

Attention  was  paid  to  presenting  a variety  of  problems  from  the 
mathematics  curriculum  strands,  and  to  encouraging  substantial  student 
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responses.  No  attempt  was  made  to  control  the  nature  of  the  problems  at 
each  grade  level,  and  teachers  prepared  a large  "bank"  of  problems  from 
which  to  choose. 

The  teacher  researchers  considered  the  format  of  the  problem  statement 
and  the  student  response  sheet  to  be  important  elements.  A format  was 
selected  for  each  grade  level,  including  a prompt  for  students  to  show  all 
their  work,  a response  structure  (especially  for  grades  5 and  8)  and  some 
statement  of  affect  from  the  student  (How  did  you  feel ...  ?).  Samples  of 
these  problems  are  found  in  Appendix  C. 

Developing  Holistic  Marking  Criteria 

The  EQI  teacher  researchers  began  the  exploration  of  scoring  problems  by 
reviewing  problem  solutions  provided  anonymously  by  other  teachers. 
These  solutions  were  used  in  several  ways:  to  identify  quality  in  student 
responses;  to  develop  some  common  expectations  for  student  problem 
responses;  and  to  gain  an  appreciation  of  the  variety  of  problem-solving 
responses  and  interpretations  there  could  be. 

The  teacher  researchers  used  marking  practices  that  varied  from  holistic  or 
impressionistic,  related  to  the  overall  nature  of  student  performance,  to 
andytic  or  rubric-based,  including  specific  predefined  requirements  that 
reflect  the  expert  opinion  regarding  the  important  elements  of  a problem. 
The  teachers  reviewed  a wide  variety  of  systems  used  in  holistic  marking 
prior  to  developing  some  formats  for  scoring  the  problems  used  in  the 
EQI-Math  project.  An  holistic  structure,  representing  each  problem 
solution  as  either  "A"  inadequate,  "B"  adequate,  or  "C"  competent  were  the 
categories  used  during  the  first  year. 

Documenting  Student  Understanding 

Teacher  researchers  continually  considered  the  problem  of  understanding 
the  thoughts  of  students  that  were  not  expressed  in  their  written  solutions. 
They  attempted  to  improve  the  fullness  of  student  responses  in  various 
ways:  by  preparing  a structured  response  form  for  students  in  grades  5 
and  8;  by  using  the  encouragement  "Please  Show  and  Explain  All  Your 
Work"  on  every  response  sheet;  and  by  providing  lots  of  paper  for  the 
student  response.  Teachers  also  considered  the  use  of  journal  entries  or 
learning  log  entries  to  clarify  student  thinking. 

The  project  activities  in  the  first  year  considered:  the  presence  in  students 
of  positive  dispositions  towards  problem  solving;  the  quality  of  math 
problems;  marking  approaches  for  the  written  responses;  and  discovering 
more  about  the  thinking  of  students.  The  activities  of  the  second  year 
built  on  this  work. 
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Year  2:  Field  Testing 


The  project  activities  in  the  second  year  addressed  five  factors  that 
appeared  to  contribute  to  quality  problem  solving  in  mathematics: 

1)  student  problem-solving  dispositions;  2)  selection  and  presentation 
of  problems;  3)  holistic  scoring  scales;  4)  student  attitudes;  and  5) 
conditions  of  mathematics  learning  in  the  classroom.  EQI-Math 
teachers  and  advisors  prepared  and  field-tested  materials  and  practices  as 
they  worked  together  to  extend  their  knowledge  about  each  of  these  five 
factors. 

Student  Problem-solving  Dispositions 

Teacher  researchers  documented  classroom  observations  of  the  10  student 
dispositions  on  several  occasions.  Students  were  observed  in  mathematics 
classes  (not  only  during  the  completion  of  the  EQI  assigned  problems)  and 
a record  was  to  be  made  of  whether  the  disposition  was  "present”  (+),  "not 
present"  (-),  or  "not  observed"  (0).  A sample  Disposition  Checklist  is  found 
in  Appendix  C. 

The  teacher  researchers  found  that  using  the  disposition  checklists  was  a 
difficult  activity.  First,  many  teachers  found  the  number  of  observations  to 
be  made  cumbersome. 

• I really  have  found  the  inventory  sheets  and  disposition  sheets  for 
the  whole  class  time  consuming  and  difficult  to  do  so  early  in  the 
school  year.  One  problem  may  show  only  one  or  two  of  the 
dispositions  given.  So  it  seemed  to  take  a lot  of  problems  in  order  to 
assess  all  dispositions. 

• [There  are]  too  many  dispositions  to  track  for  each  student. 

Second,  many  teachers  were  personally  uncomfortable  with  the  process  of 
using  the  checklist  and  the  inferences  that  were  made  from  the  observed 
behaviours. 

• 1 was  overwhelmed  and  frustrated  trying  to  complete  the  dispositions 
— I like  to  be  accurate  and  somehow  see  this  [impression]  as  being 
written  in  stone. 

• Some  of  the  dispositions  — I felt  I was  guessing  on  and  would  have 
difficulty  giving  valid  reasons  why  I had  given  the  child  a or  a 

• I feel  the  disposition  sheets  are  not  valid  at  this  point.  I'm  just 
beginning  to  know  my  students  and  it  is  only  now  that  / feel 
comfortable  making  comments  on  the  students. 

EQI  - Art  & Mathematics  43  Calgary  Board  of  Education 

Calgary  Catholic  Board  of  Education 


However,  teachers  did  find  that  the  exercise  of  completing  the  disposition 
sheets  allowed  them  to  discover  interesting  and  valuable  information 
about  individual  students. 

• Fm  finding  it  difficult  to  be  objective.  However,  1 do  find  that 

knowledge  about  the  children,  how  they  approach  tasks,  what  kind 
of  a communicator  they  are,  is  important. 

During  the  second  year,  the  teachers  were  also  provided  with  ’’Anecdotal 
Record”  sheets  to  record  observed  behaviours  for  each  of  the  10 
dispositions  for  two  students.  Each  teacher  was  asked  to  record 
observations  of  one  ’’good  problem-solving”  student,  and  one  ’’weak 
problem-solving”  student.  An  example  of  one  of  these  records.  Inventory 
of  Mathematical  Disposition  Experiences  for  ....  appears  in  Appendix  C. 

This  was  a challenging  task,  but  all  teacher  researchers  completed  at  least 
two  of  these  records  for  students  in  their  classes.  These  records  were 
collected  into  grade  group  descriptions,  and  summarized  for  students  at 
each  of  the  grades. 

The  anecdotal  record  activities  proved  to  be  very  illuminating  for  the 
teacher  researchers  and  also  for  the  project  advisors.  There  was  a variety 
of  interpretations  of  the  dispositions,  and  variation  in  the  clarity  of 
description  provided  by  the  teachers.  Some  teachers  prepared  lengthy 
descriptions  of  student  behaviours  — "Her  communication  about  doing 
math  problems  is  at  afunctional  level.  For  example,  first  you  do  this 
and  then  this.  She  doesn't  seem  to  look  for  concepts."  and  other  teachers 
recorded  very  brief  comments,  such  as  "Student  communicates  well." 

Because  identifying  student  dispositions  and  recording  descriptions 
presented  a number  of  difficulties  for  the  teacher  researchers,  some 
revisions  to  the  materials  and  procedures  were  made.  Three  approaches  to 
reformulating  the  identification  of  dispositions  were  pursued: 

a.  Consolidating  the  dispositions  to  be  identified.  Factor  analysis 
using  a principal  components  method  with  iterations  was  carried  out 
on  the  checklist  data  prepared  by  the  teacher  researchers  during  the 
second  year  of  the  project.  One  significant  factor  was  obtained 
which  was  highly  related  to  other  factors.  For  the  pilot  year,  in  order 
to  conceptualize  this  factor,  dispositions  were  reduced  to  five  and 
descriptions  altered  and  refined.  The  five  dispositions  and  their 
descriptions,  ’’Mathematical  Dispositions,”  are  in  Appendix  C. 

b.  Reducing  the  number  of  observations  required  of  teachers.  Pilot 
teachers  were  asked  to  record  observations  of  the  five  problem-solving 
dispositions  twice  during  the  pilot  period  — early  in  September  and  at 
the  beginning  of  December,  toward  the  end  of  the  pilot. 
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c.  Restructuring  anecdotal  record  forms  and  procedures.  Pilot 

teachers  were  given  inservice  with  a highly  structured  format  prior  to 
the  pilot.  Three  records  were  made  during  each  problem  assignment. 

Selection  and  Presentation  of  Problems 

The  teacher  researchers  selected  eight  problems  to  field  test  with  their  math 
classes.  The  student  responses  were  rated  by  the  teacher,  then  by  one 
other  teacher  at  the  same  grade  level.  Teachers  at  each  grade  level  met 
several  times  during  the  field  test  to  discuss  concerns  and  new  learnings. 

Field  testing  the  problems  provided  information  to  teacher  researchers  on 
the  following  issues: 

a.  Wording  of  problems.  The  teacher  researchers  discovered  that 
several  problems  were  misunderstood  by  students  partially  as  a result 
of  the  words  used.  Problems  were  reworded  by  the  advisors. 

b.  Answer  formats.  Advance  organizers,  in  the  form  of  "boxes"  for 
parts  of  the  problem-solving  process  for  students  at  grades  5 and  8 
received  mixed  reviews  from  EQI-Math  teacher  researchers  and 
advisors.  Several  teachers  found  the  boxes  very  helpful  in  organizing 
and  encouraging  the  work  of  their  students,  while  others  observed 
that  the  boxes  inhibited  student  responses.  The  advisors  decided  to 
remove  the  boxes  from  the  problem  solution  sheets  used  in  the  pilot. 
Problems  for  piloting  are  printed  on  11"  by  17"  sheets,  folded  to  a 8.5" 
by  11 " format.  The  problem  sheets  provide  about  three  regular  sized 
sheets  of  space  for  the  student  solution. 

c.  Time  allowed  for  completion  of  problems.  During  the  field  testing 
of  problems,  teachers  decided  on  the  appropriate  time  limits  in  their 
own  classes.  Although  the  time  allowed  varied  among  classes,  this 
practice  did  not  seem  to  pose  any  problems  for  the  project  and  was 
continued  in  piloting. 

d.  Individual  or  group  administration.  Teacher  researchers  selected  a 
method  of  problem  presentation  that  suited  their  classes  and  teaching 
styles.  Some  encouraged  students  to  confer  with  partners  or  in  small 
groups  and  some  required  the  students  to  work  strictly  by 
themselves.  All  students  prepared  a response  sheet  for  each  problem. 
Although  there  was  some  discussion  regarding  the  "fairness"  of  using 
collaborative  processes,  the  advisors  decided  not  to  restrict  how  a 
teacher  presented  the  problem.  The  pilot  teachers  were  asked  to 
record  if  collaboration  was  allowed  with  each  problem.  Teachers 
were  reminded  that  the  EQI-Math  project  materials  were  not  designed 
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to  compare  individual  achievement  for  evaluative  purposes,  and 
therefore  did  not  require  the  strict  control  of  conditions. 

Consultants  working  on  the  provincial  EQI  project  encouraged  the 
EQI-Math  advisors  to  review  and  revise  problems  as  necessary,  with  a view 
to  producing  a set  of  guidelines  for  the  construction  of  "good”  problems. 
Rather  than  keeping  EQI-Math  problems  confidential,  consultants 
encouraged  the  development  of  many  problems  and  the  preparation  of 
problem  construction  guidelines.  It  was  also  determined  that  six  problems 
would  be  sufficient  to  provide  qualitative  samples  during  piloting. 

Holistic  Scoring  Scales 

The  teacher  researchers  field-tested  eight  problems  at  each  grade  level. 
They  were  asked  to  write  out  their  holistic  scoring  criteria,  then  score  each 
problem  solution  either  "A"  inadequate,  ”B"  adequate,  or  ”C"  competent. 
This  letter  rating  was  marked  on  the  back  of  the  student  response.  A 
second  teacher  re-scored  the  paper,  using  the  same  three  letter  markings.  In 
some  cases,  a third  marker  was  used  to  provide  an  additional  rating  of  the 
student  response.  Teachers  did  not  discuss  their  interpretations  of  the 
holistic  criteria  at  length  prior  to  the  scoring  exercises. 

After  the  scoring  process  was  completed,  teachers  were  asked  to  explicidy 
define  their  interpretations  of  the  holistic  scoring  criteria.  These  were 
collected  and  then  summarized. 

Consultants  with  the  provincial  EQI  project  reviewed  the  holistic  scoring 
processes  and  criteria,  and  recommended  that  the  project  consider 
structuring  categories  that  reflected  levels  of  elaborated  thought  — from 
very  minimal  or  simple,  to  highly  elaborated,  abstract  conceptions. 
EQI-Math  advisors  were  encouraged  to  provide  more  than  three 
categories  of  holistic  ratings  — perhaps  by  adding  a rating  of  "elegant,"  or 
by  adopting  a five  point  scale. 

Subsequendy,  the  EQI-Math  advisors  prepared  a new  holistic  scale.  The 
characteristic  that  discriminates  between  the  first  three  levels  of  scoring  is 
the  degree  of  progress  made  toward  a solution,  from  partial  to  complete. 

The  fourth  level,  "elegant,"  is  self-explanatory.  Advisors  also  changed  the 
depiction  of  the  holistic  categories  from  scoring  criteria  to  response 
criteria.  A copy  of  the  Holistic  Response  Criteria  is  found  in  Appendix  C. 

Student  Attitudes 

In  order  to  understand  better  how  students  respond  to  problem  solving, 
the  teacher  researchers  decided  to  include  a measure  of  student  attitudes 
with  each  of  the  EQI  problems.  The  attitude  question  was  placed  at  the 
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bottom  of  each  answer  sheet.  During  the  field  testing,  each  grade  level 
used  a different  attitude  response  format. 

The  grade  2 format  included  three  faces:  "happy,"  "neutral,"  and 
"unhappy."  Data  analysis  indicated  that  student  attitude  related  to  problem 
ratings  when  the  problem  was  difficult.  However,  it  was  interesting  to 
note  that  a student  who  received  a rating  of  "A"  (inadequate)  on  a problem 
that  is  easy  or  of  mid-range  difficulty  was  just  as  likely  to  be  "happy"  as  a 
student  who  received  a rating  of  "C"  (competent).  Put  in  another  way,  a 
student  who  received  a rating  of  competent  is  just  as  likely  to  be 
"unhappy"  as  a student  who  received  a rating  of  inadequate. 

Grade  5 students  were  provided  with  the  prompt:  "How  I feel  about  this 
problem?  Explain."  with  lines  provided  for  the  response.  Grade  8 students 
were  asked:  "Circle  words  that  tell  how  you  feel  about  this  problem  or  write 

your  own  words.  Easy Challenging Confusing 

Fun or ."  At  the  Grade  10  level,  students  were  asked  to: 

"Circle  words  that  tell  how  you  feel  about  this  problem  or  write  your  own 

words.  Boring Challenging Confusing Fun 

or ." 

Asking  students  about  their  feelings  towards  a problem  was  thought  to 
help  teachers  prepare  more  interesting  or  appealing  problems.  Another 
dimension  of  student  attitude  is  the  feeling  a student  has  after  completing  a 
problem.  To  obtain  this  information,  advisors  decided  to  ask  students  to 
respond  to  a statement  that  refers  to  how  they  feel  after  attempting  to 
solve  the  problem. 

The  EQI-Math  pilot  problems  included  two  statements: 

The  word  1 would  use  to  describe  this  problem  is: 

Boring Challenging Confusing Fun 

After  trying  to  solve  this  problem  I feel: 

Discouraged Satisfied Frustrated Happy  


Conditions  of  Mathematics  Learning  in  the  Classroom 

Teacher  researchers  were  asked  to  assist  in  describing  their  classroom 
mathematics  environment  by  responding  to  the  statement:  "Describe  what 
is  happening  in  your  math  classroom."  This  description  was  to  be  made  at 
the  time  of  presentation  of  each  EQI  problem.  Each  teacher  researcher 
provided  eight  descriptions  of  the  classroom  environment.  No  specific 
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format  was  prescribed  for  the  responses,  although  teachers  were 
encouraged  to  talk  about  such  elements  as  the  curriculum  emphasis,  style 
of  teaching,  and  unique  classroom  features  they  thought  might  influence 
the  problem-solving  results. 

Some  of  the  teachers  used  the  response  to  describe  lesson  topics: 

• Geometry  has  been  the  main  focus  - symmetry,  constructing  3-D 
shapes,  2-D  shapes. 

• Free  explorations  of  manipulative s.  Group  problem  solving  every 
day.  Group  graphing  (i.e.  birthdays).  Pattern  problems  (Guess  my 
rule:  shoes  with  laces  or  without  laces). 

Other  teachers  described  student  organization: 

• We  have  formal  group  instruction  in  math  in  the  morning  (with  or 
without  manipulatives,  depending  on  the  developmental  level  of  the 
children).  In  the  afternoon  we  have  learning  centres  in  which  the 
children  practice  and  extend  their  skill  development.  The  children 
sit  in  groups  of  five.  We  have  some  group  solving  of  problems  as 
well  as  individual.  We  will  also  be  working  in  pairs,  but  have  done 
little  of  this  yet. 

Still  other  teachers  focused  on  student  responses  to  the  mathematics 
instruction: 

• Students  are  beginning  to  enjoy  problem  solving.  They  begin 
problems  in  groups  and  understand  never  to  leave  a piece  of  paper 
blank.  Some  are  becoming  real  risk-takers,  while  a few  want  to 
always  play  it  safe  and  find  the  "correct”  answer. 

• Students  are  becoming  better  able  to  cooperate  and  work  in  small 
groups  as  the  school  term  continues. 

Teacher  researcher  responses  varied  in  comprehensiveness,  style,  and 
detail.  There  was  little  in  common  among  the  responses  to  characterize 
classroom  styles. 

A very  interesting  finding  for  the  EQI-Math  advisors  was  the  indication 
that  for  some  of  the  teacher  researchers,  problem  solving  is  still  a "Friday 
event"  rather  than  an  integral  part  of  the  daily  mathematics  instruction. 

Teacher  researchers  were  asked  to  respond  to  the  statement:  "How  did 
you  present  this  problem  to  your  students?"  Again,  no  specific  framework 
was  given  for  the  responses  and  again,  both  the  quality  and  the  nature  of 
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the  records  varied  considerably.  This  information  identified  whether  or  not 
students  collaborated  with  one  or  more  classmates  while  working  on  the 
EQI  problems  and  provided  additional  background  about  the  classroom 
environment. 

After  working  with  the  EQI-Math  project  for  two  years,  the  teacher 
researchers  were  asked  to  respond  to  a set  of  questions  regarding  their 
experiences  in  working  with  the  project  and  how  the  project  affected  their 
teaching  practice  in  the  area  of  mathematical  problem  solving. 

The  teacher  researchers  were  unanimously  positive  about  the  experience 
in  working  with  the  project,  most  often  citing  the  opportunity  to  work 
with  colleagues,  the  learning  about  students,  and  the  exposure  to  new 
ideas.  Their  concerns  with  the  disposition  ratings,  anecdotal  records, 
holistic  scoring,  and  problem  solution  formats  were  used  to  edit  the 
problems  and  amend  procedures  for  the  pilot.  The  need  for  time  to  meet 
more  often  and  for  inservicing  was  a primary  concern  in  their 
recommendations . 

When  asked  about  the  impact  on  their  teaching  and  student  problem 
solving,  all  reported  some  positive  effects  of  EQI.  Some  of  their  comments 
follow. 

• / have  begun  to  experiment  with  the  introduction  of  problems 
before  the  required  skills  have  been  taught.  Hopefully,  this  will 
give  more  meaning  to  the  mathematical  concept! 

• Vm  integrating  problem  solving  into  many  math  tasks,  not  just 
treating  it  as  a strand.  It's  incorporated  into  other  subjects.  My 
observation  skills  are  improving. 

• [Students  have]  increased  confidence.  1 had  students  score  some  of 
their  own  work  holistically,  and  now  they  are  more  self-aware. 

• / believe  my  students  are  more  confident  about  problem  solving  and 
have  become  greater  "risk-takers. " They  are  very  willing  to  try  to 
solve  problems  and  use  more  strategies  than  they  did  in  September. 

Year  3:  Pilot  Testing 

The  project  activities  in  the  third  year  centred  around  the  pilot  testing  of 
materials  in  local  classrooms  and  the  gathering  of  results  from  that  pilot  test. 
This  section  reviews  the  following  activities  of  the  third  year:  1)  pilot  testing 
activities;  2)  analysis  of  pilot  testing  data;  3)  development  of  "Problem 
Solving  Profiles"  handbook;  and  4)  findings  and  conclusions. 
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Pilot  Test  Activities 


The  EQI-Math  materials  were  piloted  in  eight  Calgary  Board  of  Education 
and  Calgary  Catholic  Board  of  Education  classrooms  from  September  to 
December  1991 . Pilot  teachers  were  identified  by  the  system  mathematics 
specialists  as  being  successful,  professionally  involved  teachers  of 
mathematics.  If  school  principds  gave  their  consent,  prospective  pilot 
teachers  were  asked  if  they  wished  to  volunteer.  The  pilot  teachers  had  no 
previous  involvement  with  the  EQI-Math  project. 

Selected  pilot  teachers  attended  two  days  of  inservice  for  which  they  were 
given  release  time  from  their  teaching  duties.  The  EQI-Math  advisors  and 
a professor  from  the  University  of  Calgary  provided  sessions  covering  all 
the  aspects  of  the  pilot  process,  from  the  philosophy  of  problem  solving  to 
filling  out  the  necessary  project  forms.  This  section  includes:  a)  problem- 
solving dispositions,  b)  preparing  anecdotal  records,  c)  presenting  the 
problems,  d)  problem  rating,  e)  documenting  classroom  conditions,  and 
f)  providing  feedback  about  the  effects  of  piloting. 

a.  Problem-solving  dispositions 

Pilot  teachers  observed  their  students  at  the  start  of  the  pilot,  and 
recorded  whether  or  not  each  of  the  five  dispositions  was  ’’displayed” 
or  ’’not  displayed”  or  ’’not  observed.”  The  five  dispositions  are: 

• motivated 

• creative 

• confident 

• strategic  approach 

• strategic  process 

Pilot  teachers  recorded  a second  observation  of  student  dispositions 
during  the  administration  of  the  fifth  problem  in  the  pilot  set.  Record 
sheets  provided  to  teachers  did  not  contain  the  information  about  the 
first  teacher  observation. 

b.  Preparing  anecdotal  records 

The  pilot  teachers  practiced  preparing  anecdotal  records  during  the 
inservice  by  viewing  a videotape  of  student  problem  solving  and 
recording  observations.  Three  students  were  observed  during  each 
of  the  problem  assignments  and  records  made  on  structured  anecdotal 
records  forms.  Teachers  were  asked  to  select  one  student  they 
thought  to  be  a good  problem  solver,  one  who  is  a weak  problem 
solver,  and  an  ”I’m-not-sure-about-problem  solver.”  A sample 
anecdotal  record  appears  in  Appendix  C. 
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c.  Presenting  the  problems 

Pilot  teachers  administered  six  problems  to  the  selected  pilot  classes 
during  the  same  weeks.  Teachers  could  allow  students  to  collaborate 
with  their  peers  at  the  beginning  of  the  problem  session,  but  all 
students  were  required  to  submit  a solution  for  rating,  noting  whether 
they  collaborated  with  others  on  their  solution  sheet.  The  time 
allowed  for  each  problem  was  determined  by  the  teacher. 

Four  graduate  education  students  interviewed  two  students  at  each 
problem  administration.  The  students  were  coached  in  "flexible 
interviewing  techniques"  and  consulted  with  Dr.  O.  Chapman,  a 
mathematics  professor,  as  necessary.  Both  video  and  audiotapes 
were  taken  of  each  problem-solving  session,  and  the  audiotapes  were 
transcribed.  Excerpts  from  the  transcripts  are  included  in  the 
"Problem  Solving  ^ofiles"  to  illuminate  processes  of  student  thinking 
that  are  not  included  in  written  responses. 

d.  Problem  rating 

The  Holistic  Response  Criteria  were  examined  and  discussed  at  the 
pilot  teacher  inservice.  Pilot  teachers  were  encouraged  to  sort  the 
student  responses  to  the  EQI  problems  into  three  groups: 
"Preliminary,"  "Partial"  and  "Complete."  Following  that,  they  were  to 
review  the  "Complete"  set,  and  select  any  responses  that  were 
"Elegant." 

Teachers  rated  their  students'  responses  using  the  holistic  response 
criteria,  and  recorded  this  rating  on  a form  sent  in  to  the  project  office. 
The  problem  responses  were  then  sent  to  the  colleague  pilot  teacher 
at  the  same  grade  level  for  a second  rating.  This  pilot  teacher  then 
sent  the  second  score  and  the  student  solutions  to  the  project  office. 

e.  Documenting  classroom  conditions 

The  pilot  teachers  responded  to  questions  about  their  teaching 
practices  and  beliefs  at  the  beginning  of  the  pilot  process  and  at  its 
completion.  These  responses  formed  the  basis  of  our  analysis  of 
changes  in  teacher  attitudes  and  behaviours  as  a result  of  using  the 
EQI-Math  materials.  A copy  of  this  survey  form,  "EQI-Math  Problem- 
Solving  Practices  and  Beliefs"  is  included  in  Appendix  C. 

EQI-Math  advisors  visited  each  pilot  classroom  and  completed  an 
"Environmental  Scan."  The  scan  attempted  to  describe  the  classroom 
environment  through  a series  of  responses  to  questions  about  the 
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classroom  organization,  teacher  practice,  materials  available,  and  other 
resources. 

f. 

Providing  feedback  about  the  effects  of  piloting 

At  the  final  pilot  teacher  meeting  in  January  1992,  the  pilot  teachers 
provided  both  written  and  oral  responses  to  questions  of  interest 
regarding  the  results  of  the  project.  All  of  the  responses  were 
maintained  and  provide  a rich  source  of  qualitative  data  for  the 
preparation  of  project  reports. 

Analysis  of  Pilot  Test  Data 

a.  Problem-solving  dispositions 

Disposition  records  for  the  first  and  second  observations  were 
compared.  Data  analysis  showed  that  in  the  second  observation,  one 
or  more  dispositions  were  observed  in  64%  of  students,  an  increase  of 
16%  over  the  first  observation.  The  most  substantial  change  was 
reported  in  the  strategic  process  disposition.  For  all  grade  levels, 
correlation  coefficients  between  dispositions  and  problem  ratings  as 
measured  by  Cramer's  V were  low  to  moderate,  ranging  from  .06  to 
.38. 

b.  Preparing  anecdotal  records 

The  anecdotal  records  developed  by  the  pilot  teachers  were  reviewed 
by  Olive  Chapman,  mathematics  professor  at  the  University  of 
Calgary.  The  records  have  been  used  to  develop  materials  regarding 
observation  and  flexible  interviewing  for  classroom  teacher  use. 

In  reviewing  the  anecdotal  records  kept  by  teachers,  it  was  apparent 
that  teachers  used  a variety  of  approaches,  were  not  uniformly 
prepared  to  do  the  task  of  documentation,  and  found  the  processes  of 
drawing  inferences  and  teaching  recommendations  very  challenging. 
Some  of  these  factors  might  have  been  eliminated  if  more  time  had 
been  spent  inservicing  and  preparing  teachers  to  observe  and  record 
student  behaviour,  and  to  make  inferences  and  teaching 
recommendations  as  a result. 

c.  Student  problem  ratings,  attitudes,  and  dispositions 

The  EQI-Math  project  was  designed  as  action  research,  and  no 
research  questions  or  proposals  were  established  prior  to  the 
implementation  of  the  project.  However,  some  data  were  collected  in 
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the  process  of  completing  the  project  activities  and  these  data  were 
available  for  whatever  information  they  might  yield. 

d.  Documenting  classroom  conditions 

Prior  to  the  pilot,  the  selected  teachers  were  asked  to  respond  in 
writing  to  a set  of  questions  regarding  their  beliefs  and  practices.  At 
the  conclusion  of  piloting,  teachers  were  asked  to  respond  again  to 
the  same  set  of  questions.  The  two  sets  of  responses  were  analyzed 
to  permit  identification  of  changes  in  individud  teacher’s  comments 
over  time. 

The  EQI-Math  Advisory  Committee  reviewed  the  responses  and 
identified  changes  in  teacher  beliefs  and  practices.  Responses  to 
questions  about  problem-solving  beliefs  did  not  seem  to  change 
significantly  between  the  two  surveys.  Teachers  selected  for  piloting 
were  already  strong  teachers,  with  firmly  held  beliefs  about  learning 
and  teaching.  However,  the  pilot  teachers  did  report  some  changes  in 
their  practices  and  some  plans  for  change  in  the  future.  Some  of  their 
comments  follow. 

• The  project  has  made  me  do  more  problem-solving  lessons.  It 
also  helped  me  track  students  differently.  It  gave  me  more 
definite  ways  to  categorize  what  I saw. 

• [I  will]  be  more  conscious  of  the  process  rather  than  the  end 
product.  Give  students  plenty  of  problems  at  the  beginning  of 
the  year  and  not  mark  them  ...no  risk  involved. 

• I now  mark  holistically  because  it  gives  a clearer  picture  of  the 
strategies  students  are  using,  and  I also  have  a better  idea 
about  which  students  would  benefit  from  interviews. 

e.  Providing  feedback  about  the  effects  of  piloting 

Math  pilot  teachers  responded  in  writing  to  the  question:  "Are  your 
students  better  problem  solvers  as  a result  of  your  participation  in  the 
EQI-Math  pilot?" 

Students  were  seen  to  be  better  problem  solvers  to  some  degree  in 
every  pilot  classroom.  Pilot  teachers  provided  many  comments 
suggesting  that  improvements  occurred  as  a result  of  risk-taking, 
working  in  groups,  encouragement  to  try  different  ways,  increased 
discussion,  and  generally,  a more  positive  attitude.  Some  of  their 
comments  follow. 
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• The  encouragement  of  multiple  solutions  and  creative  responses 
allows  children  to  be  individuals,  and  respect  their  choices. 

• [There  is  a]  strong  component  of  talk  and  the  need  to  explain, 
clarify,  work  through  ...  such  that  students  developed  confidence 
to  use  their  "voice.” 

• I believe  they  are  better  now  because  Tm  looking  at  them 
through  a new  pair  of  eyes  — this  project  has  shown  me  a few 
new  things  about  how  students  attack  problems.  Thus  I can  now 
better  address  the  problems  they  may  have  and  I think  I can 
better  appreciate  why  some  are  having  difficulties  ...  hopefully, 
this  will  help  us  all  to  become  better  problem  solvers! 

The  pilot  teachers  also  discussed  their  responses  to  the  various 
aspects  of  the  EQI-Math  pilot  — the  holistic  scoring,  disposition 
identification,  anecdotal  records,  student  attitude  reports,  and 
conditions  that  encourage  quality  problem  solving.  The  summary 
notes  from  these  discussions  were  prepared  by  the  math  advisors. 

Holistic  scoring  was  well  received  by  all  teachers  although  secondary 
teachers  were  concerned  with  becoming  more  precise  and  expanding 
the  descriptors.  In  general,  the  focus  on  the  process  over  product,  the 
lack  of  a discouraging  "mark,”  and  the  sharing  of  expectations  about 
a good  solution  were  seen  as  very  positive  features. 

Disposition  identification  was  positively  received  by  most  of  the 
teachers  because  it  "focused  the  observations"  and  provided  a tool 
and  vocabulary  for  reporting.  At  the  junior  high  school  level,  there 
was  concern  that  trying  to  make  a Judgement  about  dispositions  from 
observation  was  sheer  guesswork,  and  there  was  great  variety  in 
students  from  one  problem  to  the  next. 

Anecdotal  records  were  criticized  by  all  pilot  teachers  as  being 
"frustrating,"  "time  consuming,"  "inaccurate,"  or  "interfering  with  the 
teaching  process."  A few  positive  comments  related  to  the  possible 
use  in  student  interviews. 

Student  attitude  reports  were  criticized  by  pilot  teachers  at  all  grade 
levels.  In  the  early  grades,  teachers  felt  that  the  students  didn’t  really 
understand  the  words.  In  the  senior  grades,  there  was  concern  about 
the  vocabulary  used,  and  whether  students  took  the  response 
seriously.  A few  students  at  each  grade  level  were  seen  to  give  the 
response  a lot  of  thought. 
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Conditions  that  encourage  quality  problem  solving  were  similar  for  all 
grade  levels.  Teachers  identified  time,  the  opportunity  to  work  in 
small  groups,  the  free  discussion  of  ideas  without  judgement,  a 
facilitating  teacher,  resources  in  the  classroom,  and  good  problems  as 
important  conditions. 

The  EQI-Math  teacher  researchers  and  pilot  teachers  provided  a large 
amount  of  qualitative  information  about  their  experiences  and 
observations  throughout  their  work  with  the  project.  This 
information  was  reviewed  by  an  evaluation  specialist  who  prepared  a 
summary  questionnaire  based  on  the  information  in  the  qualitative 
responses.  This  questionnaire  and  the  results  are  included  in 
Appendix  C. 

Nineteen  (79%)  of  the  24  teachers  who  had  been  involved  with  the 
project  responded  to  the  survey.  All  of  the  respondents  agreed  that 
their  participation  in  EQI  has  been  a valuable  learning  experience^ 
and  95%  agreed  that  involvement  in  EQI  has  made  them  a better 
teacher  of  mathematics.  Teachers  at  all  grades  who  participated  in 
the  EQI  project  experienced  the  greatest  increase  in  skill  and 
confidence  in  the  area  of  assessment  of  student  growth  in  problem 
solving. 

During  the  piloting  of  EQI-Math  materials,  the  mathematics  advisory 
committee  visited  all  pilot  classrooms  at  least  once,  and  prepared  an 
’’environmental  scan”  regarding  the  classroom  conditions. 

Math  advisors  expressed  discomfort  with  using  one  or  two  limited 
observations  as  indicative  of  a classroom  environment,  so  this 
information  is  not  used  in  the  project  findings. 

The  two  pilot  teachers  at  the  grade  5 level  asked  students  to  respond 
in  writing  to  their  experiences  during  piloting.  Students  were  given 
some  class  time  and  encouraged  to  contribute  their  ideas. 

The  students  reported  that  they  learned  about  approaches  to 
problems  as  well  as  interpersonal  relationships  in  doing  the  EQI 
problems.  The  student  responses  all  had  positive  elements.  Here  is 
one  sample: 

• I really  liked  doing  the  problem  solving.  It  was  very  interesting. 
I think  it  was  a good  idea  to  work  in  groups  because  if  you 
didn't  understand  the  problem,  the  group  could  help  you.  I 
learned  that  sometimes  I chose  the  wrong  strategies  and  I 
learned  that  I should  check  the  problem.  Sometimes  my  group 
did  not  make  a plan  and  we  got  the  problem  wrong. 
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Development  of  "Problem  Solving  Profiles"  Handbook 

The  teacher  researchers  and  the  pilot  teachers  reported  that  discussing 
student  responses  with  other  teachers  was  an  important  benefit  of 
participation  in  this  project.  Recognizing  the  value  of  this  sharing  of  ideas, 
the  EQI-Math  advisors  began  in  1991  to  plan  the  development  of  a 
"Problem  Solving  Profiles"  book  that  would  provide  teachers  with 
approaches  to  problem-solving  assessment  from  the  EQI  project. 

The  profiles  contain  information  about  student  dispositions,  holistic 
scoring,  and  flexible  interviewing.  For  each  grade  level,  a selection  of 
student  solutions  illustrate  samples  of  student  work  that  show  the 
characteristics  of  the  holistic  response  criteria.  Student  solutions  were 
selected  from  the  pilot  project.  Each  solution  is  accompanied  by 
observations  of  expert  math  teachers  who  have  been  involved  with  the 
EQI-Math  project. 

The  "Problem  Solving  Profiles"  handbook  contains  the  elements  described 
above.  The  handbook  presents  classroom  assessment  of  three  elements: 

1 ) what  students  do  when  solving  problems;  2 ) how  students  think  when 
solving  problems;  and  3)  what  students  write  down  in  response  to 
problems.  "What  students  do"  can  be  observed  and  recorded  in  anecdotal 
records  and  linked  to  the  student  dispositions.  "How  students  think"  can 
be  approached  through  the  use  of  flexible  interviewing  techniques.  "What 
students  write"  is  shown  in  their  problem  solutions  and  can  be  scored 
holistically.  The  handbook  provides  a small  amount  of  theory  and 
practical  examples  that  model  assessment  of  each  element  of  student 
problem-solving  behaviours. 

Findings  and  Conclusions 

The  Calgary  EQI-Math  Project  identified  indicators  of  quality  student 
work  in  problem  solving  in  mathematics,  grades  2 to  10.  Indicators  can 
inform  the  educational  practice  of  teachers  and  students  in  several  ways. 

• Teachers  can  use  EQI-Math  materials  to:  plan  and  implement 
instructional  strategies  and  programs  appropriate  to  student  needs; 
maintain  accurate,  informative  documentation  regarding  the 
educational  attainments  of  students;  and  report  back  to  students, 
parents,  administrators  and  others  about  these  attainments. 

• Students  can  compare  their  efforts  with  models  or  exemplars  of 
quality  work,  develop  goals  and  personal  standards  to  attain,  and 
apply  their  energy  to  improving  their  learning. 
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After  the  development  and  testing  of  classroom  materials  designed  to 

develop  indicators  of  student  problem  solving,  project  advisors  reported 

the  following  conclusions. 

• Teachers  reported  positive  changes  to  their  teaching  and  assessment 
practices  as  a result  of  participating  in  EQI-Math. 

• Teachers  wanted  to  meet  to  discuss  teaching  and  assessment.  The 
experience  of  working  closely  with  colleagues  was  very  positively 
received. 

• Some  of  the  materials  and  practices  developed  in  the  project  were 
seen  to  be  useful.  Specificdly: 

student  dispositions  were  helpful  in  observing  students,  and  in 
developing  understandings  about  student  thinking 

holistic  scoring  was  helpful  in  two  ways:  teachers  looked 
beyond  the  answer  to  the  processes  used  by  students;  and 
observation  and  reflection  were  encouraged 

looking  at  many  student  solutions  helped  both  teachers  and 
students  understand  and  communicate  about  the  diversity  of 
responses  and  what  quality  problem  solutions  look  like 

• Some  of  the  materials  and  practices  developed  in  the  project  were  not 
seen  to  be  useful.  Specifically: 

anecdotal  records  were  very  time  consuming  and  inconsistent  for 
use  in  large-scale  problem-solving  assessments 

student  attitude  reports  were  of  limited  value  to  teachers 

generalizations  about  classroom  conditions  that  promote  problem 
solving  could  not  be  provided  from  this  project 

• Students  seemed  to  be  more  positive  about  problem  solving,  took 
more  risks,  and  demonstrated  more  confidence  as  a result  of  the  EQI 
project. 

• Systematic  methods  for  observing  students  closely  — for  example, 
the  disposition  sheets  and  extensive  answers  — helped  teachers  with 
teaching  decisions  and  with  assessment  and  evaluation  tasks. 

• Students  responded  to  the  invitation  to  express  themselves  more  fully 
and  to  think  about  their  thinking. 
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The  project  activities  did  not  determine  to  what  degree  there  was  an 
improvement  of  quality  in  student  work  as  a result  of  the  use  of  EQI 
materials.  A further  study  would  be  needed  to  understand  more  about  this 
relationship. 

In  conclusion,  the  project  developed  some  promising  approaches  to  the 
task  of  assessing  student  problem  solving.  It  is  a good  start,  and  a basis  for 
people  to  build  from  in  developing  performance  assessment  measures.  The 
methods  of  holistic  scoring  and  identifying  dispositions  contribute  to 
broader  measures  of  assessment  that  are  useful  in  understanding  and 
reporting  on  student  achievement. 
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CHAPTER  5:  SUMMARY  AND  DISCUSSION 


Action  research  begins  without  an  end  product  in  mind.  The  Calgary  EQI 
Project  began  in  just  that  "open-ended"  way.  Along  the  way,  the 
researchers  struggled  with  choosing  from  a variety  of  alternatives, 
answered  many  questions  about  assessment  practice,  and  proposed  other 
questions  that  remain  to  be  answered.  This  chapter  contains  1)  findings  of 
the  research,  2)  processes  promoting  success,  3)  response  to  the  project 
products,  4)  future  directions,  and  5)  recommendations. 

Findings 

Art  and  mathematics  appear  to  represent  very  different  assessment 
challenges,  and  we  did  not  expect  to  find  many  commonalities  in  teacher 
practice.  However,  as  the  EQI-Art  and  EQI-Math  projects  evolved,  many 
commonalities  emerged  as  themes  and  styles  of  practice.  The 
commonalities  emerged  because  the  making  of  art  and  the  solving  of 
mathematical  problems  require  complex  thinking  processes,  and  then- 
assessment  requires  informed  professional  judgement.  The  expression  and 
development  of  professional  judgement  is  the  common  focus  of  the  project 
conclusions. 

Findings  that  are  common  to  both  EQI-Art  and  EQI-Math  are: 

• Quality  assessment  models  in  art  and  mathematics  support  teacher 
decision  making  about  teaching  and  assessment.  The  EQI  materials 
were  clearly  intended  for  use  in  assessment  practice:  as  the  project 
unfolded,  it  became  obvious  that  there  was  considerable  positive 
impact  on  teaching  practice.  This  finding  supports  the  view  of 
learning  adopted  by  the  project  — students  create  knowledge.  The 
challenge  for  teachers,  then  is  to  become  skilled  in  listening  and  in 
observing  student  work  and  students  at  work. 

• Attending  to  written  and! or  spoken  student  reflections  illuminates 
the  art  work  or  the  problem  solution.  Both  art  and  mathematics 
teachers  found  that  the  additional  written  and  verbal  explanations 
provided  by  students  were  a new  "window  to  the  mind"  of  the 
learners.  In  many  instances,  teachers  found  that  assessing  student 
outcomes  without  involving  students  to  explain  their  thinking  was 
very  difficult.  Teachers  reported  that  their  attitudes  and  behaviours 
were  changing  as  a result  of  "listening"  more  to  students.  Advisors 
suggested  that  more  attention  to  the  student  "voice"  would  be  an 
obvious  further  development  of  EQI  materials.  The  importance  of 
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language  in  student  art  making  and  problem  solving  was  continually 
emphasized  in  the  project. 

• Quality  assessment  is  a shared  experience  between  teachers  and 
students,  encouraged  by  shared  exemplars  and  common 
vocabulary.  Teachers  reported  surprise  and  delight  at  the  interest 
and  sophistication  displayed  when  students  became  involved  in  their 
own  and  peer  assessment.  Teachers  and  students  require 
opportunities  for  joining  in  discussion  and  examination  of  their 
intended  meanings  and  problem  solutions. 

• Qualitative  assessment  takes  time  since  students  must  complete  a 
complex  assignment  and  teachers  must  observe  and  respond.  The 
concern  with  not  having  time  to  do  assessment  of  this  nature  was 
expressed  more  frequently  by  secondary  mathematics  teachers  and 
was  related  to  a perceived  pressure  to  cover  a required  amount  of 
prescribed  curriculum. 

• Exemplars  in  art  and  mathematics  performance  assessment  are  a 
way  to  share  expertise  and  to  develop  a sense  of  connoisseur  ship. 

A collection  of  quality  exemplars  in  subject  areas  helps  develop  high 
and  common  expectations  for  student  outcomes  and  celebrates 
diversity  in  student  responses. 

Informed  professional  judgement  is  the  critical  component  in  the 
qualitative  assessment  of  complex  student  outcomes  in  art  and 
mathematics.  The  Calgary  EQI  Project  used  the  informed  professional 
judgement  of  expert  teachers  and  subject  specialists  to  develop  and  refine 
this  project.  The  materials  and  approaches  that  have  resulted  model  the 
practice  of  assessment  for  other  teachers  and  for  students. 

Processes  Promoting  Success 

The  involvement  of  classroom  teachers  in  all  phases  contributed  to  the 
strength  and  validity  of  the  project.  Teachers  contributed  expert 
judgement,  thoughtful  reflections  about  teaching  and  learning,  and  precise 
criticism  regarding  materials  and  methods.  Teachers  showed  how  much 
they  want  to  work  with  colleagues  on  issues  of  assessment  and  they 
reported  receiving  professional  benefits  and  personal  satisfaction  from 
collegial  work  in  this  project. 

The  involvement  and  commitment  of  two  large  school  boards  provided  a 
wealth  of  background  not  only  in  subject  expertise  but  also  in  other  areas 
of  learning,  teaching  and  assessment.  Project  teachers  valued  the 
opportunity  to  work  with  new  colleagues.  Project  advisors  all  commented 
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on  the  benefits  of  working  with  colleagues  in  similar  positions  with 
another  school  board. 

Response  to  the  Project  Products 

Teachers  and  students  have  responded  positively  to  the  materials  and 
approaches  developed  in  EQI-Art  and  EQI-Math.  Chapters  3 and  4 
describe  in  some  detail  the  positive  responses  of  teachers  to  the  activities 
and  products  developed  within  the  EQI  project.  There  is  some  evidence 
that  positive  changes  occurred  in  learning  and  teaching  as  a result  of  using 
EQI  materials. 

Presentations  regarding  the  EQI  project  have  been  very  well  received. 
Teacher  and  administrator  groups  want  copies  of  the  art  handbook  and 
mathematics  materials  to  work  with  in  their  development  of  assessment 
measures.  Business  people  support  the  focus  on  significant  outcomes  and 
quality  work.  Parents  comment  on  how  the  exemplars  help  them 
understand  expectations.  University  personnel  want  to  share  these 
approaches  with  teachers  in  training. 

Future  Directions 

Teachers  and  students  have  responded  positively  to  the  materials  and 
approaches  developed  in  EQI-Art  and  EQI-Math.  There  is  some  evidence 
that  positive  changes  occurred  in  learning  and  teaching  as  a result  of  using 
EQI  materials. 

The  development  of  the  EQI  assessment  materials  required  time:  to  discuss, 
develop,  experiment,  rethink,  assess,  evaluate,  and  to  begin  again  with  new 
ideas  that  emerge.  We  were  provided  with  support  for  three  years,  and 
made  promising  advances.  To  continue  this  developmental  process 
requires  that  more  time  be  dedicated  in  the  future. 

Recommendations 

At  the  conclusion  of  the  Calgary  EQI  Project  participants  at  all  levels 
assisted  in  the  preparation  of  recommendations  for  future  development  of 
the  project  ideas  and  materials.  The  participants  recommended  that: 

• new  assessment  processes  be  implemented  into  evaluation  and 
reporting  practices; 

• the  EQI-Art  Handbook  and  the  Problem  Solving  Profiles  be 
published  and  made  available  to  schools  within  the  CBE,  the  CCBE, 
and  to  other  school  systems  on  request; 
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• institutions  involved  in  teacher  preparation  include  a variety  of 
assessment  skills  in  teacher-preparation  courses; 

• teachers  be  encouraged  to  develop  professional  collections  of  quality 
student  exemplars  in  art  and  in  mathematics; 

• teachers  be  provided  with  opportunities  to  work  on  collegial 
assessment  of  student  work;  and 

• school  jurisdictions  be  invited  to  expand  the  EQI  project  process  to 
include  other  disciplines. 

In  conclusion,  this  project  reaffirms  the  importance  of  thoughtful  reflection 
on  the  part  of  the  students  and  teachers  in  the  assessment  of  student  work. 
It  is  also  clear  that  innovation  requires  system  support,  guidance  and  time. 
When  teachers  and  students  are  informed  by  experts,  connoisseurs  and  the 
literature,  they  have  the  knowledge  and  skill  to  engage  in  meaningful 
assessment. 
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APPENDIX  B 


EQI-ART  MATERIALS  AND  REPORTS 

Handbook  Example 

• Sample  Lesson  from  EQI  Art  Handbook 

• One  Student  Sample  from  the  Lesson 

• "Quality  Indicators  in  Art" 

Review  of  Teacher  and  Student  Journals 
EQI- Art  Pilot  Teacher  Survey 
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Lesson  1 


Grade  2 


Directed  Observations  and  Design:  Water  Motion 


The  teacher  challenged  students  to  reflect  that  environments  are  altered  by  natural  forces  and  to 
observe  that  shapes  can  suggest  movement.  Students  were  to  observe  examples  of  the  changing 
patterns  created  by  animals  moving  through  water  and  to  note  how  light  is  reflected  when  water  is 
moving.  Using  oil  pastels,  students  were  challenged  to  depict  movement  through  water.  Students 
observed  many  examples  from  art  and  from  the  teacher's  picture  file.  Students  had  a choice  of 
background  and  of  animal. 

The  teacher  expected  that  the  students  would: 

• imagine  an  animal  swimming  through  water 

• create  a two-colour  oil  pastel  drawing  showing  how  water  changes  when  an  animal  swims 
through 

• make  patters  that  show  light  and  shadow  on  the  surface  of  the  water 

• indicate  the  movement  of  the  water's  wake 

Materials:  coloured  construction  paper 

Medium:  oil  pastels 

Time:  two  45-minute  class  periods  in  the  fall  term 


Student  Reflections: 

/ drew  a dog  swimming  in  the  water.  Whatever 
is  in  the  water  is  making  a wake  coming 
behind  it.  The  wake  makes  lines  and  patterns 
in  the  water.  / wanted  blues  so  that  it  would 
go  good  and  brown  to  colour  the  dark. 


Observations: 

The  student  has  created  an  unusual  pattern  of  blues,  browns  and  white  to  show  water  movement. 
Shapes  have  much  variety  in  design,  size  and  placement.  The  sharp  contrast  between  the  pattern 
in  the  moving  water  and  the  absence  of  pattern  in  the  calm  water  creates  a dramatic  feeling  of  an 
animal  forcing  its  way  through  water. 

Repeating  diagonal  lines,  colours  and  shapes  makes  unity.  The  dark  shape  of  the  dog  placed  off- 
centre  creates  emphasis,  movement  and  contrast. 

The  handling  of  the  oil  pastel  seems  confident.  Original  work  is  12"  x 18" 
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QUALITY  INDICATORS  IN  ART 


Meaning 

To  what  degree  does  the  work  successfully  EXPRESS  THE  IDEA  OR  FEELING  the  student  intended? 


Relationships 

• UNITY:  The  essential  characteristic:  everything  fits. 

How  do  all  the  parts  of  the  work  - shapes,  lines,  colours,  textures,  values  - relate  to  one  another 
and  to  the  main  idea?  Consider  questions  of  balance,  of  warm  and  cool  colours,  etc. 

• PATTERN:  rhythms  made  by  repeating  elements. 

What  lines,  shapes,  colours  are  repeated? 

What  variations  in  repeated  elements  are  noted? 

How  do  the  repeated  elements  strengthen  the  idea/mood  of  the  work? 

• EMPHASIS:  a focus  in  the  work. 

What  shapes,  lines,  colours  are  emphasized? 

How  does  the  use  of  emphasis  support  the  idea/mood  of  the  work? 

• CONTRAST : shapes  show  up. 

How  does  the  student  use  dark  and  light,  hard  or  soft  edges  or  textures  to  communicate  the 
idea/mood? 

• ELABORATION:  significant  details. 

Where  have  details  been  added? 

How  do  these  details  strengthen  the  idea/mood? 

• MOVEMENT/SPACE:  lines,  colours,  shapes,  space  movements  create  paths  for  the  eye  to  follow. 

Is  the  space  deep  or  shallow? 

How  are  positive  and  negative  space  handled? 


Handling 

To  what  degree  does  the  work  show  confidence,  control  and  flexibility  in  the  use  of  the  materials? 

• CONTROL  OF  THE  TECHNIQUE:  student  has  mastered  the  needed  skills. 

• FREEDOM  AND  SECURITY:  use  of  medium  - paint,  chalk,  graphics,  etc.  - shows  confident  handling. 

• THOUGHTFULNESS:  work  shows  evidence  of  care  and  planning. 

• EXPERIMENTATION:  work  shows  new  uses  or  several  uses  of  a chosen  medium. 

• ABILITY  TO  USE  THE  EXPRESSIVE  CHARACTERISTIC  QUALITIES  OF  THE  MEDIUM:  softness  of 
chalk,  fluidity  of  paint,  sharpness  of  pen  and  ink,  etc.,  support  this  idea. 


EQI  Art  and  Mathematics 


B - 75 


Calgary  Board  of  Education 
Calgary  Catholic  Board  of  Education 


Educational  Quality  Indicators  Project 
REVIEW  OF  TEACHER  AND  STUDENT  JOURNALS 


Submitted  to  Jane  Sereda 
April  3, 1992 


Michelle  Zederayko 
Margaret  Latta 

Considerations 

The  consideration  of  the  EQI  journals  provides  some  insights  into  the  nature  of 
the  project  on  one  hand  and  the  difficulties  of  getting  art  students  to  write  about 
their  art  on  the  other  hand.  Use  of  the  journals  ranged  from  abstract  ramblings 
which  were  very  student-directed  to  a set  series  of  questions  which  were 
provided  by  the  teacher  and  attached  to  the  journal. 

At  the  outset,  it  was  interesting  to  note  that  each  classroom  was  notably  different 
in  approach  and  that  the  student  comments  appeared  to  reflect  the  individual 
classroom.  Student  responses  ranged  from  the  student  who  suggested  that,  "it  is 
quality  work  because  I have  put  a full  four  classes  into  the  project,"  to  the  student 
who  asserted  that,  "it  is  quality  work  because  the  shoe  looks  like  the  shoe  I was 
supposed  to  draw."  One  obscure  reference  to  the  quality  Indicators  Handbook 
suggested  that  the  student's  work  was  quality  work,  "because  I compared  them 
and  they  were  almost  the  same."  It  is  assumed  that  the  comparison  was  between 
the  student  work  and  the  work  in  the  Handbook,  although  this  is  not  made  clear 
in  the  journal. 

The  suggestion  that  being" almost  the  same"  meant  that  quality  had  been 
produced  raises  some  concerns  about  the  role  of  the  Handbook  in  the  classroom. 
A student  producing  facsimile  works  of  art  because  he/she  believes  it  is  a sure-fire 
way  to  produce  quality  is  a frightening  prospect  and  one  which  merits  further 
attention.  It  suggests  that,  in  order  for  the  Handbook  to  serve  a useful  function  in 
the  classroom,  its  use  needs  to  be  highly  mediated  by  a teacher  who  is 
himself/herself  comfortable  with  the  idea.  Classrooms  where  the  teacher  journal 
suggested  a high  degree  of  anxiety  (ostensibly  due  to  the  fact  that  the  teacher 
did  not  feel  that  he/she  was  enough  of  an  "expert")  also  resulted  in  student 
journals  which  tended  to  be  very  lockstep  in  their  consideration  of  the  notion  of 
quality  in  their  art.  Classrooms  where  the  teacher's  and  student's  journals 
suggested  a high  degree  of  reflection  resulted  in  student  comments  which  did  not 
appear  to  consider  the  Handbook.  One  such  student,  who  was  discussing  the 
degree  of  completion  in  his/her  picture  said,  "When  I look  at  a picture  I ask  myself 
is  it  really  done,  and  to  answer  that  I compare  the  image  in  my  head  to  my 
picture." 
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It  is  hard  to  accurately  assess  the  merits  of  the  program  in  the  light  of  journals 
from  students  in  the  lower  elementary  grades  since  the  task  of  writing  became  a 
real  impediment  to  an  accurate  recording  of  the  artistic  process.  Since  the 
concept  of  reflective  practice  in  the  art  making  process  is  inherently  good,  this 
problem  could  possibly  be  circumvented  through  use  of  a tape  recorder.  We  feel 
that  the  value  of  dialogue  as  a means  to  sense-making  cannot  be  overstated. 

Perhaps  the  most  significant  implication  from  a study  of  the  journals  was  that 
even  at  the  elementary  levels,  there  was  a notable  difference  in  both  student  and 
teacher  response  between  classes  where  the  teacher  appeared  knowledgeable 
and  classes  where  the  teacher  appeared  to  be  doing  his/her  best  with  limited  art 
training.  Teachers  with  art  backgrounds  provided  their  students  with  a more 
extensive  vocabulary  of  art  terms  and,  to  judge  by  the  kinds  of  comments 
appearing  in  their  students’  journals,  a concurrent  understanding  of  basic  art 
fundamentals.  The  students  in  classes  where  the  teacher’s  journal  reflected  a 
strong  art  background  showed  a far  greater  inclination  to  use  their  journals  as 
workbooks  which  contained  plans  for  future  works,  as  well  as  thumbnail 
sketches  and  insightful  discussions  of  past  works. 

Conversely,  student  journals  from  classrooms  in  which  the  teacher’s  journal 
entries  suggested  a lack  of  comfort  with  the  subject  area  produced  substantially 
different  journals.  Many  of  these  students  wrote  very  little  in  their  journals  and 
what  they  did  write  appeared  to  be  very  teacher-directed,  albeit  with  little  use  of 
an  extensive  art  vocabulary.  That  is,  these  students  appeared  to  write  in  their 
journals  primarily  in  direct  response  to  questions  posed  by  the  teacher  and  their 
responses  often  did  not  reflect  the  same  degree  of  informed  thought  about  the 
nature  of  the  artistic  process. 

Thus,  the  journals  would  appear  to  suggest  that  the  EQI  Handbook  becomes  a 
viable  teaching  tool  in  the  hands  of  a teacher  who  has  a fairly  high  degree  of 
comfort  in  teaching  art.  The  journals  would  suggest  that  a teacher  with  little  art 
experience  is  inclined  to  be  either  threatened  by  the  Handbook,  or  to  use  the 
Handbook  as  a type  of  Art  Recipe  book. 

Teacher  comments  indicated  a desire  for  more  extensive  inservicing  and  support 
for  using  the  Handbook,  and  several  pointedly  suggested  that  the  actual  "how 
to"  of  using  the  Handbook  effectively  in  their  classes  was  a difficult  practical 
concern.  One  teacher,  who  saw  the  Handbook  as  a selection  of  "model  lessons," 
pointed  out  that  more  information  on  teacher  motives  should  accompany  each 
project  in  the  Handbook.  Another  teacher  raised  a cautionary  note  in  regard  to 
the  use  of  the  Handbook  as  the  start  of  a project  since  it  "stifled  the  creative 
process  because  they  (the  students)  think  that  it  is  the  only  model." 

Clearly,  the  EQI  Handbook  has  some  exciting,  as  well  as  sobering,  implications  for 
its  use  in  the  classroom.  The  implication  which  requires  the  most  thought, 
however,  would  appear  to  be  the  fact  that  no  handbook,  no  matter  how  good  or 
well-thought-out,  can  replace  expertise  on  the  part  of  the  teacher. 
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Implications 


The  Art  Making  Experience 

During  art  making  or  creating,  many  students  wrote  in  their  journals  of  being 
involved  with  the  medium.  They  described  art  making  as  a "thinking,”  "working," 
"constructing,"  "changing,"  "building"  process.  Seemingly,  the  art-making 
experience  required  a relationship  between  the  artist  and  the  medium.  It  was  a 
search  for  qualities  - the  perception,  selection,  and  organization  of  qualities  and  a 
responsiveness  to  them.  The  use  of  the  journals  by  art  teachers  and  art  students 
held  the  possibility  for  engaging  them  in  thinking,  feeling,  reflecting,  and  talking 
throughout  the  art-making  process.  The  facilitation  of  this  dialogical  sense- 
making process  was  the  strength  of  the  journal  use  by  teachers  and  students. 

Some  journals  evidenced  progressively  much  more  depth  and  insight  than  others. 
Acknowledging  that  there  were  many  factors  that  could  have  contributed  to  this, 
we  felt  generally  that  students  who  had  been  encouraged  to  involve  themselves 
in  the  art-making  process,  valuing  this  process  over  the  product,  and  where  an 
ongoing  dialogue  was  promoted,  evidenced  greater  understanding  in  their 
journals.  We  surmised  that  the  dialogic  sense-making  process  was  dependent  on 
involvement,  dialogue  and  community.  These  are  intertwined  and  dependent  on 
each  other  as  the  art  form  was  created.  Above  all,  the  teacher's  place  in  making 
this  happen,  was  central. 

Involvement:  Inquiry  Guided 

Individual  interpretation  should  be  encourage.  Art  making  is  a search  that  starts 
with  the  self.  Students  that  were  encouraged  to  ’’play"  with  a medium 
discovering  for  themselves  the  nature  of  the  medium  and  a solution  of  their  own 
making,  made  comments  in  their  journals  that  reflected  greater  sense  making  and  a 
tacit  knowing  developing  . . . "slowly  learning  more  and  more  about  drawing";  "I 
decided  not  to  add  another  colour  there  because  I felt  it  would  be  a distraction"; 

"I  said  no  to  myself  because  I decided  it  would  be  too  much."  These  same 
students  seemed  far  less  concerned  with  the  end  product  and  more  interested  in 
critically  reflecting  on  their  own  work  and  others'  in  progress. 

Dialogue  and  the  Place  of  Community:  A Discourse  by  Nature 

A large  part  of  the  search  process  is  the  ongoing  dialogue  generated  between  the 
emerging  art  work  and  the  artist.  The  attention  drawn  by  teachers  to  quality 
indicators  seemed  to  encourage  students  to  see  the  interrelationships  of  elements 
with  in  a whole.  It  also  helped  students  find  the  words  to  verbalize  their 
reflections  on  their  own  and  their  peers'  art  work.  This  discourse  was  greatly 
facilitated  by  teachers  who  responded  to  their  students'  work  in  a manner  that 
continued  the  conversation  rather  than  shutting  it  down. 
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In  Conclusion 


We  saw  the  journals  as  holding  the  possibility  for  being  an  excellent  means  to 
foster  artistic  learning.  The  opportunity  for  teachers  to  reflect  on  their  own 
practices  was  described  by  many  as  "invaluable.”  Their  constant  questioning  and 
striving  to  understand  the  project  was  made  clearer  through  their  reflections, 
observations  and  feelings  about  their  art  classes  and  students.  We  saw  the  place 
of  the  teacher  as  pivotal  in  regards  to  facilitating  the  use  of  the  journals  in  an 
approach  that  promoted  a continuing  and  evolving  discourse.  The  students' 
journals  fostered  reflection  on  their  art  learning,  a heightened  sensitivity  to 
elements  within  a whole  developed.  It  is  hoped,  therefore,  that  this  encouraged 
quality  improvement  in  student  art  work. 
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EQI-ART  PILOT  TEACHER  SURVEY 

PURPOSE  OF  THE  SURVEY:  This  survey  is  designed  to  gather  information  about  the  art 
assessment  beliefs,  practices  and  concerns  of  teachers  in  the  CBE  and  CCSB  in  order  to 
develop  assessment  programs  and  materials  for  Art  teachers.  No  individual  responses  will  be 
used  in  reporting,  and  you  need  not  identify  yourself  on  the  questionnaire. 

If  you  have  questions  or  comments  about  the  survey,  please  contact  Jane  Sereda,  Educational 
Quality  Indicators  Project,  Calgary  Board  of  Education,  telephone  294-8243. 

10  out  of  11  pitot  teachers  (91  %)  recoded  to  the  survey 

1.  1990-91  ART  TEACHING  ASSIGNMENT 


Division  I (Grades  1-3) 

Average 

1.5 

hours  per  week  Art  instruction 

Division  II  (Grades  4-6) 

Average 

6.5 

hours  per  week  Art  instruction 

Division  III  (Grades  7-9) 

Average 

8.0 

hours  per  week  Art  instruction 

Division  IV  (Grades  10-12) 

Average 

15.0 

hours  per  week  Art  instruction 

2.  NUMBER  OF  YEARS  TEACHING  EXPERIENCE  - Write  the  number  of  years  you 
have  taught  in  each  Division  on  the  line  below.  The  mixed  Division  category  is  for 
teachers  who  have  had  teaching  assignments  across  Divisions. 

E.C.S  Div.I  Div.I/II  Div.II  Div.II/III  Div.III  Div.III/IV  Div.IV 


TOTAL  YEARS  "TEACHING”  

Bei^iidents  reported  from  3 to  2S  years  of  teaching:  the  average  vms  14^  years* 

3.  NUMBER  OF  YEARS  OF  EXPERIENCE  TEACHING  ART  - Write  the  number  of  years 
you  have  taught  Art  in  each  Division  on  the  line  below. 

E.C.S  Div.I  Div.I/II  Div.II  Div.II/III  Div.III  Div.III/IV  Div.IV 


TOTAL  YEARS  "TEACHING  ART"  

Average  jmmber  of  years  ^‘teaching  art**  was  12*3  yearsr 

4.  NUMBER  OF  UNIVERSITY  OR  COLLEGE  ART  COURSES  TAKEN  (one  full  course 
and  one  half  course  would  be  written  as  "1.5"  courses) 


Art  Methods 

Average 

2.4 

courses 

Art  History 

Average 

2.4 

courses 

Art  Studio 

Average 

8.0 

courses 

other  Art  Courses  (specify): 

courses 
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5. 


In  the  past  two  school  years  (since  September,  1989)  about  how  many  hours  have  you 
spent  directing  in  art  inservices  and/or  giving  professional  assistance  related  to  the 
teaching  or  the  assessment  of  art? 

Average  21.2  hours 

Range  from  10  - 50  hours 


6.  For  each  of  the  following  descriptors,  please  circle  the  number  that  represents  the 
degree  to  which  each  describes  YOU: 


very  descriptive 

a practising  artist  1 2 

a specialist  teacher  (in  Art)  n = 8 1 2 

a non-specialist  teacher  (in  Art)  n = 2 1 2 


not  descriptive 
3 4 5 

3 4 5 

3 4 5 


7.  If  you  had  your  choice  of  teaching  assignments,  would  you:  (Circle  the  ONE  number 
that  best  represents  your  choice.) 

a.  express  a preference  to  teach  Art  - 9 out  of  10  rospondonts 

b.  express  a willingness  to  teach  Art  - 1 out  of  10  respondents 

c.  express  a preference  not  to  teach  Art 

If  you  answer  "c,"  please  explain  why: 


8.  Beliefs  about  Art  instruction  and  assessment  are  illustrated  in  the  statements  that  follow. 
Please  read  and  respond  to  each  of  the  following  statements  by  circling  the  letters  that 
indicate  your  level  of  agreement  with  each  statement  on  the  scale: 


All  responses  from  this  point  on  reflect  ”after  pilot”  results. 


a. 

b. 


c. 


d. 


SA  = Strongly  Agree 
A = Agree 
U = Undecided 
D = Disagree 
SD  = Strongly  Disagree 

Art  instruction  has  a significant  effect  on  a student’s 
ability  to  produce  good  art  work. 

It  is  important  for  all  students  to  have  quality  Art 
instruction  as  part  of  their  public  school  education. 
Innate  talent  determines  how  well  students  perform 
in  Art. 

The  identification  of  quality  work  in  Art  is  based  on 
individual  taste. 


SA 


U 


D SD 


9 1 

2 6 2 


3 7 
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e. 


f. 

g- 

h. 


j- 


All  students  are  capable  of  developing  good  skills 
in  Art  depiction  and  composition. 

Art  courses  should  be  given  the  same  level  of 
importance  as  all  other  courses  in  the  school 
curriculum. 

Objective  criteria  should  be  the  basis  for  assessing 
quality  work  in  Art. 

It  would  be  helpful  for  teachers  to  have  common 
criteria  for  assessing  the  quality  of  student  art  work. 
Art  courses  should  be  provided  with  a similar 
amount  of  resource  support  as  are  other  courses  in 
the  school. 


4 3 12 

2 8 

9 1 


Checklist  are  more  valid  than  individual  critiques  in  2 6 2 

assessing  student  art. 


There  was  yery  little  change  In  these  answers  before  and  alter  piloting. 


9.  The  statements  below  represent  impressions  of  the  status  of  Art  in  a school.  Please 
indicate  your  level  of  agreement  about  how  well  each  statement  is  representative  of  the 
general  feelings  IN  YOUR  SCHOOL  on  the  scale: 


a. 

b. 


c. 


d. 

e. 

f. 

g- 

h. 


SA  = Strongly  Agree 
A = Agree 
U = Undecided 
D = Disagree 
SD  = Strongly  Disagree 

Art  is  an  important  subject. 

All  students  should  take  Art  from  Grades  1 to  6. 

All  student  should  take  Art  at  least  once  in 
secondary  grades. 

Art  is  a source  of  pride  for  the  school. 

The  objectives  of  the  Art  program  are  understood. 
The  Art  program  turns  students  on. 

Art  is  highly  visible  in  the  school. 

The  Art  program  receives  its  share  of  resources  and 
support. 

At  our  staff  meetings,  Art  receives  the  same 
attention  as  other  school  subjects. 


SA 


U D 


SD 


4 3 3 

3 5 1 

4 3 12 


5 4 

3 3 3 

5 5 

7 1 1 

4 


1 3 2 3 1 


There  was  almost  no  change  in  responses  before  and  after  pilotingr 
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1 0.  Below  is  a list  of  assessment  methods  some  teachers  use  in  Art  programs.  For  each  of 
the  methods  listed,  please  indicate  how  often  you  use  each  method  in  your  current  Art 
classes  by  CIRCLING  the  appropriate  term  on  the  right. 

Except!  where  noted,  there  wa$  almost  no  change  In  responses  before  and  after 
piloting* 


written  tests 

less  after  piloting 

checklists 

portfolios 

visual  journals  or  sketchbooks 

written  critiques  of  individual  art 
works 

verbal  critiques  of  student  work 
student  self-ratings 

slight  increase  after  piloting 

peer  ratings 

slight  increase  after  piloting 

numeric  scores  on  art  handed  in 

observations  of  work  habits 

other  methods  I use  in  assessing 
Art 


ALMOST 

ALWAYS 

QUITE  A 
LOT 

SOME- 

TIMES 

SELDOM 

ALMOST 

NEVER 

1 

9 

1 

1 

4 

2 

2 

6 

3 

1 

1 

3 

4 

2 

1 

2 

5 

1 

3 

6 

1 

2 

4 

2 

2 

1 

3 

1 

1 

4 

4 

3 

3 

2 

2 

1 
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1 1 . Below  are  various  categories  teachers  use  in  marking  student  achievement  in  Art 
courses.  Beside  EACH  OF  THE  CATEGORIES,  indicate  the  percentage  that  category 
contributes  to  the  report  card  mark  you  give  students.  If  you  DO  NOT  use  the 
category,  please  put  in  a ZERO.  The  sum  of  aU  entries  should  be  100%. 


Range 

Mode 

student  knowledge  of  art  theory 

0 - 50  % 

0 

student  artwork  produced 

16  - 50  % 

20,  50 

attitude  to  the  subject 

2 - 20  % 

10 

attendance 

0 - 12  % 

0 

willingness,  cooperation 

0 - 20  % 

10 

group  work 

0 - 13  % 

0 

creativity/originality 

1 - 30  % 

20 

completion  of  project  on  time 

0 - 15  % 

0 

other  factors  (specify): 


There  was  little  change  in  responses  befbre  and  after  piloting*  A small  change  was 
noted  in  slightly  more  emjf^asis  placed  on  ’^artwork  produced**  and  slightly  less 
emphasis  on  **wiillngne^,  cooperation  and  attitude’*  after  piloting* 
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12.  Below  is  a list  of  factors  that  may  play  a role  in  how  you  plan  your  Art  lessons.  Circle 
the  number  that  indicates  the  level  of  importance  each  factor  has  on  your  practice. 

There  were  no  slgniUcant  changes  in  responses  before  and  after  piloting. 


a 

E 

o 

> 


a 


o 

o 

c 

iS 

b. 

a 

E 

C4-1 

O 


s. 

E 

E 

P 


integration  with  other  subjects 

the  Art  curriculum  guide 

the  Art  Resource  Guides 

recommendations  from  school 
administrators 

personal  collection  of  successful  lessons 

perceived  student  needs 

recommendations  from  colleagues 

recommendations  from  system  art 
specialists 

cultural  celebrations  in  the  school 
cultural  celebrations  in  the  community 
seasonal  art  (leaf  art  in  fall) 
my  own  judgement  as  a teacher 
other:  (specify) 


4 2 13 

3 3 4 

3 3 3 

2 3 4 1 

3 5 2 

5 4 

3 4 2 1 

4 2 3 

3 1 5 

2 15  1 

2 2 4 1 

6 3 1 
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14. 


What  are  the  limitations  you  see  in  delivering  a quality  art  program  in  your  school? 


Thank  you  for  your  cooperation. 
September,  1991 
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APPENDIX  C 


EQI-MATH  MATERIALS  AND  REPORTS 


Year  One: 

Mathematical  Dispositions 

EQI-Math  Year  One: 

Sample  Problems 

Year  Two: 

Disposition  Checklist 

Year  Two: 

Inventory  for  Sample  Student 

Year  Three: 

Mathematical  Dispositions 

Year  Three: 

Holistic  Response  Criteria 

Year  Three: 

Sample  Anecdotal  Record  Form 

Year  Three: 

Problem-Solving  Practices  and  Beliefs 

Survey  of  Teacher  Participants  in  the  EQI-Math  Project 
Math  Problems  used  in  Piloting 
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EQI-Math  Year  One 


Mathematical  Dispositions  and  Observable  Characteristics 


Perseverance 

reluctant  to  stop,  willing  to  persevere 
in  mathematical  task* 
tries  different  methods* 
not  easily  discouraged 
works  to  a goal 

completes  work  (even  when  not 
assigned  for  homework) 
works  for  a lengthy  period  of  time  on 
one  strategy 

Risk-Taking 

not  afraid  to  make  mistakes  (several 
trials  and  errors) 

flexible  in  exploring  mathematical 
ideas  and  trying  alternative  methods* 
asks  questions 

positive  attitude  toward  trying  new 
things 

Confidence 

confident  In  using  mathematics  to 
solve  problems,  to  communicate 
ideas,  and  to  reason* 
confident  that  an  answer  will  be  found 
sure  of  correctness  even  when 
explanation  is  difficult 

Motivated 

actively  doing  something  (asking, 
charting,  computing) 
shows  interest,  curiosity  and 
inventiveness  in  doing  mathematics* 


Achieving 

sets  goals 

successful  in  many  areas  of  schooling 
values  recognition  for 
accomplishments 


Shows  Understanding 

knows  the  meaning  of  the  words  in 
the  question 

can  state  the  problem  in  their  own 
words  or  picture  or  model 
knows  what  the  question  asks  for 
has  some  idea  of  the  form  the  answer 
will  take 

recognizes  a reasonable  answer  and 
discards  an  unreasonable  answer 

Communicative 

provides  verbal  or  written  explanation 
of  response 

uses  mathematical  language* 
talks  through  problems  with  others 
shares  ideas,  piggybacks  on  the  ideas 
of  others 

Creative 

own  problems,  formulates  problems* 
enjoys  finding  more  than  one  solution 

Strategic/Organized 

looks  at  problems  from  different 
directions 

uses  logical  approaches,  classifies, 
estimates 

sorts  data  to  see  patterns 
applies  mathematics  to  situations 
arising  in  other  disciplines  and 
everyday  experiences* 

Reflective 

inclined  to  monitor  and  reflect  on  their 

own  thinking* 

learns  from  mistakes 

appreciates  the  role  of  math  in  culture, 

as  a tool  and  as  language* 


* Indicates  terminology  used  in  Curriculum  and  Evaluation  Standards  for  School 
Mathematics,  NCTM  (1989) 
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EQI-MATH  YEAR  ONE:  SAMPLE  PROBLEMS 


EQI-2 

Problem  #1 


Teacher  # 

Name  

Date 


SHOW  AND  EXPLAIN  ALL  YOUR  WORK 

Mary  Jane  saw  5 rabbits. 

How  many  ears  did  those  rabbits  have  in  all? 


Mart  an  X on  ^ face  that  shows  how  you  feel  about  hiis  problem. 
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EQI-5 


Teacher  # 

Problem  #1  Name  

Date  

SHOW  AND  EXPLAIN  ALL  YOUR  WORK 

The  sum  of  my  digits  is  2. 

I’m  greater  thii  12. 

Who  am  I? 

Planning  Board  (What  do  we  know?  What  do  we  want  to  find  out?) 


Doing/Drawing  Board  (Soludon,  Diagram,  Chaxt,  Giid) 


How  I feel  about  this  problem 
Explain 
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EQI-8 


Problem  #1 


Teacher  # 

Name  

Date  


SHOW  AND  EXPLAIN  ALL  YOUR  WORK 


Margie  is  a blonde.  Rose  Mary  is  a redhead,  and  Shirley  is  a brunette.  They  are  mamed  to  Alex. 
Frank  and  John,  but 

(a)  Shirley  does  not  like  John 

(b)  Rose  Mary  is  married  to  John's  brother 

(c)  Alex  is  married  to  Rose  Mary's  sister 

Who  is  mamed  to  whom?  (Assume  that  married  people  like  each  ocher.) 


Understand 


Easy  Challenging  Confusing  Fun  or 
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EQMO 


Problem  #1 


Teacher  # 

Name  

Date  


SHOW  AND  EXPLAIN  ALL  YOUR  WORK 

1 X 1 paving  stones  aie  used  to  make  a border  around  a rectangular  pond.  In  the  example  below 
the  pond  is  3 x 9 and  28  stones  are  used. 


II 1 II 1 1 1 

Pond 

- _L  _L 

What  are  the  different  dimensions  of  ponds  that  would  require  36  paving  stones  to  complete  the 
border? 


Circle  words  that  tell  how  you  feel  about  this  problem  or  write  your  own  words 
Boring  Challenging  Confusing  Fun  or 
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EQI-MATH  YEAR  TWO:  DISPOSITION  CHECKLIST 


Student  Name 

-- 

V 

Confidence 

Perseverance 

Risk  Taking 

Understanding 

Achievement 

Orientation 

Motivation 

Communication 

Creativity 

Organization 

Reflection 
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EQI-MATH  YEAR  TWO:  INVENTORY  FOR  SAMPLE  STUDENT 


What  student  has 
demonstrated: 


1.  Confidence  in  using 
mathematics 


Grade  2 Date  and  Activity 

#15  Nov.-  Dez./90 


- low  risk  wish  new  ideas/concepts 

- confident  on  ^ tasks 

- no  confidence  in  problem-solving  tasks,  always  seems 
unsure  what  to  do/how  to  proceed 


Perseverance  at  - not  at  problem-solving  tasks  - sits  and  looks  around, 

mathematical  tasks  needs  to  be  prompted 

• Low  perseverance  level  - shows  more  avoidance 


3.  Risk-taking  in  doing  - prefers  st^e,  htown  type  of  tasks 

mathemancai  tasks  • no  confidence  in  new  ideas  - needs  to  be  prompted 

- won’t  even  really  try  - prefers  to  copy  partner  or  tries  to 
avoid  doing  the  task 


4.  Understanding  about  > appears  to  have  a good  basic  understanding  (noted 

mathematics  dwing  calendar  time),  yet  d^erences  arise  in  new 

siruanons 

- lack  of  confidence  affects  performance 


5.  Achievement  orientation 
towards  mathematics 


6.  Motivation  to  do 
mathematics 


- enjoys  Math 

- does  tasks  willingly  - ceraers 

- gives  impression  of  Wang  Math 


1.  Communication  about  • able  to  verbalize  a task  he  htows,  but  usually  sky, 
mathematics  unwilling  to  diviiss 

- when  he  is  unsure  or  doesn’t  htaw , shrugs  shoulders 
7 don’t  lotow" 


8.  Creativity  in  doing 
mathematics 


- not  obvious/not  creative 
• too  busy  trying  to  avoid  the  task 


9.  Organization  in  doing 
mathematics 


. organized  when  he  understands  the  problem 
- rdies  on  others  to  give  chtes  what  to  do  though 


10.  Refiective  about  • not  really  reflective 

mathematical  thinking  . doesn’t  appear  to  reflect  on  what  he  is  doing  or  what 

he  needs  to  do  in  a task 
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EQI-MATH  YEAR  THREE:  MATHEMATICAL  DISPOSITIONS 


Motivated 

shows  interest,  curiosity,  enthusiasm  and  inventiveness  in  doing 
mathematics* 

goes  beyond  the  assignment 
reluctant  to  stop* 
works  to  a goal 

works  until  satisfied  with  solution 

works  for  a lengthy  period  using  one  strategy 


• Creative 

flexible  in  exploring  mathematical  ideas* 

tries  alternative  methods* 

invents  own  problems,  formulates  problems 

able  to  improvise  in  selecting  materials  to  represent  the  problem 


• Confident 

belief  in  ability  to  use  mathematics  to  solve  problems* 

belief  in  ability  to  reason* 

not  afraid  to  make  mistakes 

belief  that  problems  can  be  solved 

ability  to  identify  and  use  necessary  resources 

prepared  to  demonstrate  for  other  students 

belief  in  ability  to  communicate  ideas 


• Strategic  Approach 

can  state  the  problem  in  own  words 

understands  the  meaning  of  words  in  the  problem 

is  able  to  draw  a diagram  that  reflects  the  problem 

knows  what  information  is  needed  to  solve  the  problem  and  what 

information  is  extraneous** 


• Strategic  Process 

uses  organized  and  logical  approaches 
takes  time  to  stop  and  think 
notices  and  corrects  one's  mistakes** 
monitors  one's  thinking*  - is  reflective 


* Terminology  used  in  NCTM,  Curriculum  and  Evaluation  Standards  for  School  Mathematics,  1989.  p. 
228 

**  Terminology  used  in  Alberta  Education,  Diagnostic  Mathematics  Program,  1991,  "Cognitive 
Processes." 
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EQI-MATH  YEAR  THREE:  HOLISTIC  RESPONSE  CRITERIA 
Preliminary  Response 
• No  attempt 

- Words  do  not  reflect  the  problem; 

- diagrams  and  drawings  unclear  or  misrepresent  the  problem;  and/or 

- does  not  indicate  which  information  is  appropriate  to  finding  a 
solution. 

Begins,  but  fails  to  move  toward  a solution. 


Partial  Response 

• Begins  the  problem  in  a manner  that  could  lead  to  a solution  but  fails  to 
complete  it; 

omits  a significant  part  of  the  problem; 

makes  a major  computational  error; 

uses  an  inappropriate  strategy  for  solving  the  problem. 

• Provides  a solution,  but  explanation  unclear; 

argument  or  explanation  incomplete; 
diagram  is  inappropriate  or  unclear. 

(A  correct  answer  with  no  evidence  to  support  the  solution  must  be 
further  supported  by  an  interview  or  else  there  is  no  basis  on  which  to 
judge  the  process.) 


Complete  Response 

• Gives  a complete  response  that  includes  a full  solution  with  reasonably  clear 
explanations; 

communicates  sound  supporting  arguments; 

shows  understanding  of  the  mathematical  ideas  and  processes. 


Elegant  Response 

• Gives  a complete  response  with  an  "elegant”  explanation; 
unusual  insight  displayed  in  solution; 
communicates  strong  supporting  arguments. 
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EQI-MATH  YEAR  THREE:  SAMPLE  ANECDOTAL  RECORD  FORM 


ANECDOTAL  RECORD  Student  No.  51 8 

t«i.c  Problem  #1 


Observations: 

Appears  to  be  reading  her  paper. 

Draws  a rabbit  erases  it 
Three  rabbits  drawn. 

Bending  over,  looking  under  her  chair. 

Looks  around  at  people  in  her  area 
Tapping  pencil  on  table. 

Looks  around. 

Draws  more  rabbits,  a different  view. 

Writes  words. 

Interpretation: 

Not  dear  on  what  she  needs  to  do. 

I Sees  others  drawing  rabbits  and  follows. 

Once  they  are  drawn  she  is  not  dear  on  where  to  go  next 

Recommendations: 

Speno  more  time  helping  her  to  understand  what  the  problem  means,  what  is  the  question 
asking. 

Needs  more  teacher  drected  model  working  through. 


mt  1MI-C 


ANECDOTAL  RECORD 
Problem  #2 


Student  No.  518 


Observations: 

Writing  numbers,  drdes  some  numbers  -1  23456789  10. 

Gets  art  eraser. 

Erases  all  numbers. 

Draws  picture  of  girt 

CouTrts  girts  - 5 • adds  more. 

Interpretation: 

She  didn't  understand  the  problem. 

She  felt  it  should  be  numbers  but  then  went  back  to  draw  a picture  because  she  is  familiar 
with  that 

Recommendations: 


Model  a simBar  problem  for  her. 

Talk  her  through  the  words  to  be  sure  that  it  is  being  understood  (ESL  student)  • greater? 
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EOl-Mifli 
Piot  1991-82 


ANECDOTAL  RECORD 
Problem  #4 


Student  No. 


EQI  - Art  & Mathematics 


C - 100 


Calgary  Board  of  Education 
Calgary  Catholic  Board  of  Education 


PROBLEM  SOLVING  PRACTICES  AND  BELIEFS 

Responses  at  Beginning  and  End  of  Piiot 
October  1991  and  January  1992 

8 Respondents 

• HOW  DO  YOU  ADDRESS  PROBLEM  SOLVING  IN  YOUR  MATHEMATICS  PROGRAM? 
Pilot  Teacher  #1 


Before  Pilot  After  Pilot 


Not  very  well!  Try  to  give  problems  each  week, 
related  to  material  being  studied;  different  from 
regular  text  work.  We  work  through  the  solving 
once  students  have  attempted  them  - ideas  for 
strategies  from  students. 

1 try  to  model  thinking  processes.  "This  is  what 
should/might  trigger  this  approach." 

References  made  to  Polya's  approach  - on  wall 
in  classroom. 

Hot  much  change  here  - no  time  due  to 
incorporation  of  new  M30  curriculum  (attention 
paid  elsewhere).  Polya  model  on  class  wall  - 
attention  directed  to  model  as  appropriate 
throughout  units.  Teacher  model  - as  go 
through  homework,  lecture.  Student  sharing  of 
ideas  - good/not  so  good  approaches 
highlighted  (from  homework,  problem  solving, 
questions). 

Pilot  Teacher  #2 

Before  Pilot  After  Pilot 

Students  are  made  aware  that  the 
mathematical  principles  and  concepts  taught 
are  mere  tools  to  help  people  share  problems. 
We  set  aside  one  period  a week  for  problem 
solving.  These  problems  are  not  from  the  text 
but  chosen  from  other  sources  that  have 
problems  of  a more  stimulating  nature.  We 
make  sure  that  throughout  the  year  problems 
requiring  different  strategies  are  used. 

Problem  solving  is  done  in  conjunction  with 
each  topic  so  that  students  can  see  how  the 
topic  can  be  applied.  Sometimes  1 start  a new 
unit  with  a problem  that  requires  the  new  topic 
to  be  taught,  1 find  the  students  see  the  need 
for  what  is  being  taught  as  well  as  generates 
ideas  on  the  new  topic.  Our  tests  have  one- 
third  of  the  questions  in  problem  form.  This 
shows  the  importance  we  attach  to  problem 
solving. 

Pilot  Teacher  #3 

Before  Pilot  After  Pilot 

Problems  are  presented  at  different  times,  in 
different  modes.  A concept  can  be  presented 
and  developed  with  problems  reinforcing  and 
extending  the  concept.  Some  problems  have 
been  given  as  an  investigation  to  begin 
teaching  concepts.  Others  are  given  at 
random,  often  unrelated  to  concept  being 
taught. 

Students  have  an  opportunity  to  discuss  what 
problem  solving  is,  possible  strategies  are 
outlined,  and  other  contributing  factors  are 
shared  (number  of  solutions,  ways  to  get 
solution).  Problems  are  presented  to  students 
with  support  and  encouragement  to  work  in 
cooperating  groups.  Cooperation  and  group 
dynamics  is  discussed.  Problems  are 
debriefed  with  discussion  including  all  topics. 
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Pilot  Teacher  #4 


Before  Pilot 

After  Pilot 

Problem  solving  is  integrated  into  all  aspects  of 
math  as  well  as  most  other  subjects. 

Problem  solving  is  integrated  into  many  parts  of 
our  day  - not  just  math.  "Math  ” problems  are 
initially  presented  with  children  "acting  out"  the 
problem.  After  repeated  "acting  out, " it  is  then 
"labelled"  with  numbers,  eventually  objects 
(blocks,  beans,  etc.)  replace  the  children 

Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

1 do  not  have  a regular  "scheduled" 
mathematics  problem-solving  time.  Many 
opportunities  arise  naturally  that  lend 
themselves  to  working  through  a problem. 
These  problems  are  not  always  "mathematical 
problems"  but  are  worked  through  in  a similar 
way.  The  advantage  of  this  is  that  the  children 
are  naturally  engaged  because  it  comes  from 
their  working  situations. 

/ do  not  have  a defined  "problem-solving"  time 
except  during  the  pilot.  There  are  many 
occasions  in  math  where  a lesson  which  is 
traditionally  taught  can  be  turned  into  a 
problem-solving  activity.  To  practise  basic 
facts  to  20  they  would  search  for  patterns  and 
share  with  the  large  group.  1 have  students 
work  on  standard  and  non-standard  type 
problems  and  discuss  "how"  each  group  solved 
the  problem. 

Pilot  Teacher  #6 


Before  Pilot 

After  Pilot 

Problem  solving  is  introduced  as  a unit  in 
September  (first  week).  At  this  time,  a method 
of  solving  problems  is  introduced.  The  problem 
solving  is  then  integrated  into  the  other  strands 
of  the  curriculum  throughout  the  year. 

Problem  solving  is  approached  as  a group 
project.  Students  work  in  groups  to  discuss 
and  prepare  answers  to  problems.  Problem 
solving  is  included  in  the  units  taught 
throughout  the  year. 

Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

1 teach  a strategy  then  give  examples.  1 
discuss  the  Houghton  Mifflin  daily  problems.  1 
use  the  problem-solving  cards  (tags). 

/ teach  a strategy  but  next  time  1 will  give  the 
problem  first,  let  them  discover  the  strategy, 
then  find  solutions.  Done  weekly,  then  similar 
problems  daily. 
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Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

Students  are  given  about  2-3  problems  each 
week.  Initially  we  solved  problems  together  as 
a class,  and  then  1 introduced  a marking 
scheme  that  1 planned  to  use  for  problem 
solving.  If  students  gave  only  the  correct 
answer,  then  they  would  receive  1 mark.  Using 
simpler  cases  and/or  making  a table  and 
spotting  patterns  -2  marks;  and  2 marks  for 
finding  a general  rule  and/or  explaining  why 
their  rule  worked. 

Students  work  on  problems  based  on  the 
current  topic  as  well  as  2 to  3 problems  per 
week,  which  may  include  logic  problems,  or 
problems  involving  the  observations  of 
patterns.  Students  are  required  to  solve  the 
problems,  and  provide  an  explanation  about 
their  solution  to  the  problem. 

SOME  PEOPLE  BELIEVE  THAT  ALL  STUDENTS  CAN  BE  TAUGHT  TO  BE  GOOD 
PROBLEM-SOLVERS.  PLEASE  COMMENT. 

Pilot  Teacher  #1 


Before  Pilot 

After  Pilot 

Good  problem  solvers  for  their  ability 
level.  If  a method/model  is  available  (a 
framework  of  reference)  I believe  all  students 
would  recognize/demonstrate  some  increase 
in  their  ability(ties)  to  problem  solve.  (They 
would  benefit  if  properly  taught  "how"  - just  like 
teachers!!) 

/ agree.  1 believe  any  student  can  "pick  up”  new 
thoughtsAechniques  through  practise  and 
through  teamwork  which  can  aid  in  the  problem- 
solving process.  These  new  thoughts 
(processes)  snowball  with  each  experience  in 
problem  solving.  Students  recall  what’s  been 
tried  (or  done)  in  the  past  and  try  to  apply  it  to 
the  new  (even  if  not  correct  approach  - it  at 
least  gives  them  a starting  point). 

Pilot  Teacher  #2 


Before  Pilot 

After  Pilot 

Problem  solving  is  not  isolated  to  math.  All 
students  have  solved  problems  in  their  daily 
lives  and  how  successful  they  are  indicates 
how  happy  they  are.  1 try  to  show  them  by 
using  more  personal  problems  how  Polya's 
model  works  and  that  they  can  be  successful  in 
the  math  area  just  as  they  are  in  other  areas. 

Yes  - all  students  can  become  good  problem 
solvers.  It  requires: 

• confidence  in  one's  ability 

• not  to  be  afraid  of  trying  something  new 

• the  basic  skills  he  or  she  needs  are 
understood. 

Problem  solving  is  how  they  will  use  math  later 
on.  1 found  with  a lot  of  patience  and 
understanding,  one  can  make  a student  a good 
problem  solver. 
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Pilot  Teacher  #3 


Before  Pilot 

After  Pilot 

True.  Students  need  to  be  given  experience  in 
solving  a variety  of  problems  with  development 
and  modelling  of  strategies  as  part  of  the 
process.  Once  strategies  are  understood, 
practice  and  experience  are  needed. 
Consideration  needs  to  be  given  to  motivation, 
sustaining  interest,  acceptance  of  challenge 
but  not  making  process  a competition. 

Students  need  guidance  and  support  accom- 
panied by  an  opportunity  to  solve  problems. 
Through  activities,  students  learn  possible 
strategies,  work  using  the  process  and  begin  to 
question  their  knowledge  and  what  to  do. 
Teachers  can  support,  assist  and  give 
opportunities  to  students  to  become  good 
problem  solvers. 

Pilot  Teacher  #4 

Before  Pilot 

After  Pilot 

1 believe  that  most  students  can  be  taught  to 
be  good  problem  solvers. 

/ believe  the  students  can  be  taught  the 
strategies.  However,  I’m  not  sure  they  can  be 
taught  to  be  creative  and/or  confident. 

Pilot  Teacher  #5 

Before  Pilot 

After  Pilot 

1 see  problem  solving  linked  to  thinking,  not  all 
people  are  effective  thinkers.  1 do  feel  with 
active  involvement,  through  time,  people  can 
change  their  view  of  themselves  as  problem 
solvers.  It  is  a way  of  thinking. 

/ would  now  say  that  some  students  can  be 
taught  to  be  better  problem  solvers. 

Pilot  Teacher  #5 

Before  Pilot 

After  Pilot 

1 agree,  if  all  students  are  given  a method  to 
approach  problem  solving  they  can  become 
good  solvers  - not  brilliant  ones! 

/ agree.  All  students  can  be  taught  to  be  good 
problem  solvers.  The  key  is  building  their 
confidence  through  exposing  them  to  varied 
problem  types  and  strategies,  and  allowing 
them  the  time  to  develop  this  confidence. 

Pilot  Teacher  #7 

Before  Pilot 

After  Pilot 

They  can  learn  some  strategies.  The  slower 
ones  need  the  most... 

We  can  teach  the  steps,  but  the  student  must 
be  self-motivated  and  needs  to  internalize  the 
strategy.  The  trying  of  a strategy  needs  more 
structure. 
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Pilot  Teacher  #S 


Before  Pilot 

After  Pilot 

1 believe  that  if  students  are  exposed  to 
various  strategies,  and  a wide  variety  of 
problems,  they  will  begin  to  think  of  problems 
as  puzzles  and  will  be  able  to  appreciate  the 
challenge,  and  experience  the  thrill  and 
satisfaction  when  a solution  is  finally  found. 

/ believe  that  all  students  can  be  taught  to  be 
good  problem  solvers.  This  can  be  done  by 
having  students  solving  problems  in  groups, 
sharing  ideas  and  strategies,  and  feeling 
comfortable  that  there  is  not  one  way  to  solve 
any  problem.  Students  will  realize  that  although 
their  approach  may  be  different,  they  could  also 
find  a solution  to  the  problem. 

HOW  DO  YOU  FACILITATE  THE  DEVELOPMENT  OF  THE  PROBLEM-SOLVING 
PROCESS  IN  YOUR  CLASSES? 

Pilot  Teacher  #1 


Before  Pilot 

After  Pilot 

Not  much  has  been  done  here. 

Sharing  student  processes  with  the  entire 
class.  Giving  suggestions  as  to  ways  of 
approaching  problems  (in  general). 

Pilot  Teacher  #2 


Before  Pilot 

After  Pilot 

We  solve  problems  in  groups  - this  leads  to 
critical  discussion  of  (a)  what  is  the  problem; 

(b)  a variety  of  possible  solutions  can  be 
suggested;  (c)  which  solution  works  or  does 
not  work  and  why;  (d)  what  process  is  required 
to  put  each  plan  into  action;  (e)  how  reasonable 
will  the  answer  be. 

Any  problems  we  have  (not  only  math)  are 
solved  using  the  problem-solving  method  - 
example,  like  preparing  for  a test,  or  door- 
decorating for  certain  activities.  After  a while 
you  find  they  tend  to  adopt  this  method  for  a lot 
of  personal  things  the  students  do  and  when  it 
comes  to  math  the  method  is  well-entrenched 
and  they  find  it  better. 

Pilot  Teacher  #3 


Before  Pilot 

After  Pilot 

Sharing  strategies.  Working  through  process 
with  students  to  develop  risk-taking,  open 
sessions  for  discussion  (build  talk  for 
understanding)  and  create  sense  of  sharing 
rather  than  working  in  isolation. 

Students  need  to  express  their  knowledge, 
become  comfortable  in  discussing  and  writing 
about  ideas,  learn  to  discuss  what  and  when 
they  don’t  know,  have  models  to  share  their 
strategies  and  realize  they  are  working  on 
problem  solving  as  a process.  Opportunity 
needs  to  be  given  to  practise  problem  solving 
and  sharing  successes,  misinterpretation  and 
all  the  real  feelings  and  happenings  as 
problems  are  worked  on. 
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Pilot  Teacher  #4 


Before  Pilot 

After  Pilot 

Divergent  type  questions  that  are  open-ended 
and  allow  for  many  possible  responses  are 
used  (e.g.  what  would  happen  if...?  How  many 
ways  can  you...?  Can  you  think  of  different 
ways? 

By  using  the  process  in  as  many  areas  of  our 
daily  classroom  life  as  possible. 

Pilot  Teacher  #5 

Before  Pilot 

After  Pilot 

Problem  solving  is  daily  life  in  a year  two 
classroom.  Problem  solving  in  math  develops 
as  we  work  through  problems  together  and 
recognize  that  there  are  many  ways  to  get  an 
answer.  1 am  conscious  of  "step-in  problem 
solving"  and  strategies  so  1 involve  the 
students  in  a wide  range  of  experiences  and 
through  the  year  to  introduce  these. 

The  discussion  after  the  students  have  worked 
out  a problem  is  important.  They  see  how 
others  did  it.  They  rethink  how  they  did  theirs. 
Students  then  work  on  a similar  problem  and 
can  change  the  way  they  did  it  first.  Materials 
and  conversations  are  encouraged. 

Pilot  Teacher  #6 

Before  Pilot 

After  Pilot 

Modelling  a successful  method  (Polya's). 
Reward  the  use  of  a successful  method.  Allow 
group  (triad)  problem  solving. 

Group  work  facilitates  discussion  of  strategies 
and  methods  to  solve  problems. 

Pilot  Teacher  #7 

Before  Pilot 

After  Pilot 

Students  work  alone,  in  pairs,  in  threes. 

The  acceptance  of  all  ideas  by  me.  1 try  to  do 
problems  too.  1 share  my  discoveries  too.  Itell 
them  1 confer  with  others.  1 don’t  use  an  answer 
key. 

Pilot  Teacher  #8 

Before  Pilot 

After  Pilot 

1 try  to  select  a variety  of  problems,  which  I 
believe  would  be  of  interest  to  my  students. 
Some  may  require  experiments,  or  working  with 
concrete  materials  to  see  patterns  and 
eventually  solve  the  problems.  Hints  are 
sometimes  given  to  the  whole  class,  or  1 may 
work  with  students  individually,  asking  guided 
questions. 

Problems  are  given  regularly.  Students  are 
given  the  opportunity  to  work  in  groups  and 
share  ideas  and  strategies  with  each  other. 
Students  are  encouraged  to  read  their  solutions 
to  the  class,  and  solved  problems  are  displayed 
in  the  classroom. 
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HOW  DO  YOU  ASSIST  YOUR  STUDENTS  TO  GAIN  INSIGHT  INTO  THEIR 
PROBLEM-SOLVING  STRATEGIES? 


Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

Using  ideas  of  strategies  to  try  via  other 
students  is  the  only  insight  used  so  far. 

1 gave  a "debriefing”  session  (10  minutes)  in  the 
class  following  the  problem-solving  hour  to 
share  what  1 was  seeing  with  all  the  students 
(their  approaches)  plus  how  1 tried  to  solve  the 
problem.  (What  was  good,  not  so  good,  why 
some  things  were  better.)  Also  posted  good 
solutions  at  back  of  class. 

Pilot  Teacher  #2 

Before  Pilot 

After  Pilot 

After  working  in  groups  - each  group  reports  on 
his  solution  - we  then  discuss  this  as  a class 
and  comment  upon  the  solution. 

They  work  in  groups  - allows  for  different 
strategies  to  come  forward.  Groups  present 
solutions  to  class  - again  show  different 
strategies  other  groups  used. 

Pilot  Teacher  #3 

Before  Pilot 

After  Pilot 

Students  begin  to  understand  strategies  and 
their  need  in  lessons  specific  to  strategy 
development.  Through  practice  and 
discussions,  students  share  their  strategies 
with  others  and  increase  ways  of  developing 
more  strategies. 

Illustrate,  model  and  discuss  strategies,  what 
processes  are  involved  and  pose  questions  to 
nudge  in  many  different  directions. 

Pilot  Teacher  #4 

Before  Pilot 

After  Pilot 

Through  questioning  - asking  for  explanations. 

Through  discussion  and  idea  exchanges. 

Pilot  Teacher  #5 

Before  Pilot 

After  Pilot 

1 have  them  share  their  method  of  solving  and 
listen  to  others'  ways.  Explaining  how  one  has 
"arrived"  at  the  answer  gives  many  places  to 
engage  them  in  thinking  about  their  thinking. 

/ ask  questions  too  about  what  they  are  doing. 
1 try  to  make  links  to  other  problems  they  have 
worked  on. 
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Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

Ask  them  to  explain  how  they  reached  an 
answer,  or  to  explain  why  their  answer  is 
reasonable. 

Providing  them  with  various  problems  that 
utilize  different  strategies.  Always  asking  them 
if  a problem  can  be  done  another  way. 

Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

They  explain  an  answer  to  the  class. 

"Tell  me  what  you  know." 

1 have  them  make  up  problems  for  the  class. 

/ always  ask  (not  tell)  "What  did  you  try?"  "Can 
you  try....?" 

Pilot  Teacher  #8 


Before  Pilot 

After  Pilot 

Sometimes  1 would  list  the  different  methods 
used  to  solve  a problem,  or  1 would  get  a 
student  to  explain  the  strategy,  he/she  used, 
then  ask  other  students  to  explain  theirs  if  it 
was  different  from  the  previous  one  read. 

/ assist  my  students  to  gain  insight  into  their 
problem-solving  strategies  by  having  different 
strategies  described  by  students,  by  displaying 
different  ways  in  which  a problem  may  be 
solved  and  by  encouraging  students  to  find 
other  ways  to  solve  problems. 

HOW  DO  YOU  MODEL  PROBLEM  SOLVING  FOR  YOUR  STUDENTS? 
Pilot  Teacher  #1 


Before  Pilot 

After  Pilot 

1 "talk  out"  the  problems;  e.g.  with  exponent 
questions  "What  do  1 see?  What  do  1 think? 
How  do  1 progress?"  1 use  them  to  think/be 
aware  of  their  thinking  processes  and  verbalize 
and  write  (in  journals). 

Through  verbal  communication  and 
demonstrations.  This  is  what  1 tried...  This  is 
what  1 was  thinking...  This  is  how  1 kept  track  of 
my  thoughts  and  plans...  etc. 

Pilot  Teacher  #2 


Before  Pilot 

After  Pilot 

1 take  a problem  that  has  nothing  to  do  with 
math  and  1 go  through  all  the  stages  in  Poly's 
model  with  them  in  solving  this  problem. 
Example:  Inviting  a limited  number  of  people  to 
your  birthday  party. 

Doing  problems  with  them  - asking  questions 
that  bring  out  the  model  to  be  used.  Use  this 
method  for  any  problems  we  have  as  a class 
(making  a team  for  sports  events). 
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Pilot  Teacher  #3 


Before  Pilot 

After  Pilot 

By  working  through  problems  with  students. 
Presenting  problems  that  do  not  have 
solutions.  Brainstorming  ideas  by  sharing  and 
accepting  suggestions.  Often  times  problems 
are  sent  home  and  parents  invited  to 
participate. 

Discuss  everything  and  anything.  Work  on 
problems  with  students.  Facilitate  activities. 
Support  and  guide  students  as  they  work. 
Pose  many  varied  questions. 

Pilot  Teacher  #4 


Before  Pilot 

After  Pilot 

Verbalize  a problem  1 have  - walk/talk  my  way 
through  to  possible  solutions. 

By  using  problems  that  arise  in  my  life  as  a 
teacher  in  the  classroom. 

Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

1 often  think  out  loud  while  working  through  a 
problem.  They  see  me  struggle  with  "how  to" 
and  self-correct  a redirect  focus;  also  that  this 
takes  time  and  help  from  others  is  appreciated. 

/ always  encourage  students  to  think  of  another 
way  to  solve  everyday  problems.  When  1 ask,  in 
math,  can  we  solve  this  another  way,  they  see 
it  as  part  of  the  lesson.  1 try  to  help  them  see 
problem  solving  is  a life  skill  and  is  flexible. 

Pilot  Teacher  #6 


Before  Pilot 

After  Pilot 

Presenting  problems  to  the  class  - solving  as  a 
group  - using  guided  questions.  Always  asking, 
"Is  there  another  way?" 

Classroom  demonstrations  by  myself  and  other 
students. 

Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

1 like  to  show  them  patterns  1 have  learned. 
"Look  what  1 found  out." 

/ do  a problem  solution  with  them. 

Pilot  Teacher  #3 


Before  Pilot 

After  Pilot 

Whenever  students  ask  questions  about  any 
problem,  1 would  usually  draw  a picture,  or  set 
up  a table,  then  ask  questions  based  on  the 
diagram  or  the  table,  to  try  to  get  students  to 
see  relationships,  and  spot  patterns. 

/ sometimes  show  students  how  to  solve  a 
problem  using  my  strategy  of  drawing  a 
diagram,  or  using  a simpler  case  based  on  the 
problem,  and  looking  for  patterns.  1 also  read  a 
problem  several  times  until  1 understand  what  is 
being  asked,  and  1 list  information  given.  1 do 
not  hesitate  to  answer  questions  based  on  the 
approach  1 am  using,  and  to  explain  why  1 used 
that  method.  1 also  acknowledge  methods  that 
are  different  from  the  one  1 applied  to  the 
problem. 
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HOW  DO  YOU  EVALUATE  STUDENT  PROBLEM  SOLVING? 


Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

1 try  to  use  a scale  of  5 (holistic  marking).  Very 
little  has  been  done  by  me  in  this  area. 

With  difficulty?  Followed  the  0-1-2  approach  of 
preliminary,  partied,  complete  response  (and 
elegant)  but  still  had  difficulties... 

Pilot  Teacher  #2 

Before  Pilot 

After  Pilot 

1 look  for  (1)  does  the  student  understand  the 
problem;  (2)  did  he  make  a plan  and  follow  it 
through;  and  (3)  how  original  is  his  solution. 

See  last  sheet. 

Pilot  Teacher  4t3 

Before  Pilot 

After  Pilot 

Student  evaluation  is  based  on  observation 
and  interaction  in  class  -contributions  to 
discussion,  participation,  interest  and  work 
within  the  process,  ability  to  acquire  the 
language  and  use  it. 

By  observation,  contributions  during 
discussion,  knowledge  and  feedback  in  oral 
and  written  form,  strategies  as  shared  during 
debriefing  sessions. 

Pilot  Teacher  #4 

Before  Pilot 

After  Pilot 

When  they’re  working  with  concrete  material  1 
observe  and  record.  Later  when  they're 
working  at  the  symbolic  and  abstract  stage,  1 
rely  on  student  products. 

Through  observation  and  some  student 
product. 

Pilot  Teacher  #5 

Before  Pilot 

After  Pilot 

1 currently  do  not  have  a specific  way  to 
evaluate  problem-solving  growth  and  change.  1 
did  watch  and  note  good  strategies,  who  gets 
frustrated,  who  sticks  with  it,  who  needs  more 
clarification  than  average. 

The  way  they  reached  their  answer  is  what  1 
look  at.  Also,  how  they  talk  about  getting  the 
solution.  1 also  like  to  see  what  they  do  when 
they  get  stumped. 
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Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

With  a focused  holistic  scoring  point  scale  (0  to 
4). 

/ use  a four-point  system: 

0 - blank; 

1 - correct  answer,  no  work; 

2 - correct  strategy  but  incomplete, 

inappropriate  strategy  - incorrect  answer; 

3 - appropriate  strategy  but  wrong  answer; 

4 - correct  strategy  and  clear  explanation 

Pilot  Teacher  #7 


Before  Pilot 

After  Pilot 

Worth  4 points:  1 plan,  1 trying,  2 correct 
answer.  Make  a problem  = 4 points.  1 0%  of 
math  mark. 

a)  0-1  points  for  each  part  understand 
strategy,  answer,  explain 

b)  Problem  of  the  week  - a candy 

c)  Tests  on  similar  patterns 

Pilot  Teacher  #8 


Before  Pilot 

After  Pilot 

Correct  answer  only  - 1 mark 
2 marks  if  strategy  was  shown 
(diagram/table)  and  recognizing  pattern  or 
relationships,  and  2 marks  for  explaining 
why  the  rule  works/drawing  conclusions 
from  the  diagrams/pattems. 

/ evaluate  student  problem  solving 
holistically,  by  giving  1 mark  for  a 
Preliminary  response:  2.5  marks  for  a 
Partial  response;  4 marks  for  a Complete 
response;  and  5 marks  for  an  Elegant 
response. 

IS  THERE  ANYTHING  ELSE  YOU  WOULD  LIKE  TO  ADD? 
Pilot  Teacher  ifl 


Before  Pilot 

After  Pilot 

1 have  done  little  problem  solving  in  my 
classroom  (with  "word"  problems).  1 wish  to 
benefit  from  materials/ideas  from  others  since  1 
don't  have  the  time/direction  to  establish 
materials  myself.  I’m  open  to  trying  something 
new!  1 look  forward  to  the  wealth  of 
knowledge  1 will  gain  from  this  project! 

If  students  gave  a complete  problem-solving 
approach  and  answered  the  question,  even 
though  the  answer  was  incorrect,  1 still  gave 
them  "Complete  Response”  rating  but  1 
constantly  struggled  with  that. 
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Pilot  Teacher  #2 


Before  Pilot 

After  Pilot 

/ found  the  journal  writing  on  the  students 
picked  very  hard  - as  / was  not  privileged  to 
what  they  were  actually  thinking  of  -but  merely 
based  my  opinions  on  observations  which  at 
times  are  misleading. 

Pilot  Teacher  #3 


Before  Pilot 

After  Pilot 

Problem  solving  needs  to  be  c»nsidered  not  a 
separate  area  of  development  but  an  integral 
process  of  program.  Many  teachers  group 
problem  solving  with  strands  for  development 
rather  than  as  a process. 

Teachers  and  students  often  need  to  break  the 
bonds  of  searching  only  for  a solution.  Key 
areas  of  development  the  teachers  and 
students  need  to  collaborate  on  are  group 
dynamics,  cooperative  learning,  metacognition, 
and  the  ability  to  realize  how  you  think  to  use 
your  abilities  most  effectively  (even 
recognizing  what  you  don't  know),  problem- 
solving strategies  and  process. 

Word  problems  are  difficult  to  use  at  the  Grade 
2 level  as  often  it  is  not  the  problem-solving 
dispositions  you  end  up  evaluating,  but  rather 
the  student's  reading  ability. 

Pilot  Teacher  #4 


Before  Pilot 

After  Pilot 

The  very  nature  of  learning  mathematics  is 
problem  solving.  We  learn  math  so  we  can 
function  in  life  and  that  functioning  is  more  than 
figuring  word  problems.  If  math  is  taught  from  a 
problem-solving  perspective,  then  we  are 
teaching  math  for  life  and  students  learn  it 
because  it  Is  meaningful  and  they  understand  it 
well  as  its  value. 

This  project  has  made  me  do  more  formal 
problem-solving  lessons.  It  also  helped  me 
track  students  differently.  It  gave  me  a more 
definite  way  to  categorize  what  1 saw. 

Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

1 thought  the  anecdotal  records  were  valuable 
but  I'd  rather  be  involved  by  mediating  the 
student’s  view  of  the  problem. 

Time  is  the  most  important  factor  in  problem 
solving.  Each  child  must  be  given  time  by  you, 
encouraging  and  guiding  them  to  carry  out  the 
problem-solving  process.  They  also  need  time 
to  develop  a method. 
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Pilot  Teacher  #5 


Before  Pilot 

After  Pilot 

1 teach  a strategy  lesson  once  a week  and  give 
sample  questions.  The  other  days  1 supply 
cards,  etc.,  for  the  groups.  Before  report  cards 
1 give  sample  tests. 

/ try  to  emphasize  "Which  strategy  do  you  plan 
to  try?  Which  strategy  did  you  use?"  to  try  to 
get  them  to  develop  a plan. 

Pilot  Teacher  #7 

Before  Pilot 

After  Pilot 

1 am  continually  looking  for  new  strategies  and 
interesting  problems  which  would  help  to  make 
my  students  better  problem  solvers. 

/ would  also  like  to  include  interviews  into  the 
evaluation  process  for  problem  solving. 
Interviews  will  help  the  students  to  clarify  the 
strategies  used,  and  enable  the  students  to 
give  a more  complete  solution  to  the  problem. 

• CAN  YOU  IDENTIFY  ANY  WAYS  YOU  HAVE  CHANGED  YOUR  TEACHING  OR 

ASSESSMENT  PRACTICES  AS  A RESULT  OF  PARTICIPATING  IN  THE  EQI-MATH 
PILOT?  PLEASE  EXPLAIN  BELOW. 

Pilot  Teacher  #1 


After  Pilot 

I see  the  benefit  of  private  interviews  to  discuss  student  progress  on  an  individual  basis  but  still 
don’t  see  how  to  fit  it  in. 

I continue  to  model  thinking  processes  (and  planning,  acting  out  plan,  etc.,  stages). 

I have  not  changed  teacher/assessment  practises  yet,  but  EQI  has  opened  my  eyes  to 
possibilities  that  I'd  like  to  incorporate  - now  I must  figure  out  how. 

The  assessment  scales  are  good  - quick  and  easy  in  general  - a checklist  would  be  most 
appropriate. 

I found  the  6 one-hour  sessions  were  too  precious  in  a semester  to  really  "give  up"  and  still  would 
prefer  a more  "inherent"  approach  in  the  Math  1 0 program  (re:  problem  solving)  - though  how?  I 
must  still  try  to  figure  out. 

Thank  you  for  the  opportunity  to  be  a part  of  this  project.  I enjoyed  learning!  Hopefully,  I can  now 
put  things  into  practice  (slowly  but  surely). 


Pilot  Teacher  #2 


After  Pilot 

There  are  some  new  wrinkles  in  my  approach  now: 

1 . Letting  the  students  read  the  problem  aloud  to  themselves,  as  well  as  to  someone  else, 
seems  to  help  them  to  understand  the  question  better.  Also,  the  interaction  of  the  two  leads 
to  better  comprehension  and  more  ideas  (strategies). 

2.  Marking  problems  using  the  holistic  approach  was  better  received  by  the  student.  They  find 
the  results  more  encouraging  and  are  apt  to  try  where  before,  they  were  afraid  to.  Makes 
them  better  risk-takers. 

3.  Has  given  me  a better  insight  on  how  they  approach  problems  and  what  areas  of  difficulty  we 
must  work  on. 
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Pilot  Teacher  #3 


After  Pilot 

The  project  brought  a problem-solving  focus  to  my  class  by  supplying  me  with  problems.  I felt  a 
need  to  explore,  investigate  and  practise,  more  immediately,  much  of  what  had  been  discussed  in 
class  prior  to  the  project.  I was  also  looking  for  more  innovation  to  work  with  students.  The  area 
of  metacognition  arose  and  became  very  useful  for  both  me  and  the  students.  For  students  who 
have  not  had  opportunities  to  discuss  and  build  understanding,  talk  assists  and  improves  their 
ability  to  formulate  Ideas  and  express  them  in  written  form.  The  project  increased  my  awareness 
and  knowledge  of  group  process,  group  dynamics,  problem-solving  process,  and  the  use  of 
critical  thinking  within  a realistic  context.  The  students  made  progress,  would  identify  and 
discuss  what  they  were  about.  Sounds  like  progress  for  me  and  the  kids!!! 


Pilot  Teacher  §4 

After  Pilot 

I believe  I do  a better  assessment  now  by  using  the  mathematical  dispositions  as  a checklist 
guide. 


Pilot  Teacher  #5 

After  Pilot 

I would  like  to  include  a problem-solving  time  where  students  record  their  "thinking"  as  in  pilot. 
This  was  helpful  in  assessment.  I can  see  advantages  in  working  on  all  types  of  problems.  I am 
sure  I have  missed  many  good  problems  and  teaching  opportunities.  The  students  may  benefit 
from  a more  consistent  "time"  to  practise  problems  in  a formal  way.  It  gives  a wide  range  of  what 
has  come  up  in  a natural  way.  I’ll  have  to  try  it  and  see.  I would  continue  to  use  the  Holistic 
Response  Criteria  to  sort  responses.  It  allows  you  to  see  strengths  and  weaknesses. 


Pilot  Teacher  #5 


After  Pilot 

I have  realized  that  discussion  in  a necessary  part  of  the  learning  process  in  problem  solving. 
Students  must  vocalize  their  thoughts  in  order  to  evaluate  their  appropriateness.  It's  somewhat 
like  working  at  the  concrete  level,  students  must  hear  it  to  understand  it.  Another  important  thing 
that  I've  learned  is  that  children  need  time  to  digest  the  problem-solving  model.  Some  children 
might  take  hours  and,  if  you  try  to  speed  up  the  process  or  cut  it  back,  you  will  inhibit  their  ability 
to  learn. 

I would  begin  problem  solving  with  lots  of  problems,  talking  and  very  little  writing. 

Be  more  conscious  of  the  process  rather  than  the  end  product.  Give  them  plenty  of  problems  at 
the  beginning  of  the  year  and  not  mark  them  (no  risk  involved). 


Pilot  Teacher  #7 

After  Pilot 

I will  spend  more  time  on  looking  for  strategy  problems.  I will  vary  the  strategies  more  in  shorter 
units  than  repeat  the  units. 
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Pilot  Teacher  #S 


After  Pilot 

As  a result  of  participating  in  the  EQI-Math  pilot,  I have  changed  some  of  my  teaching  and 
assessment  techniques.  I use  a period  per  week  for  problem  solving  in  groups,  as  well  as 
assigning  2 to  3 problems  each  week  for  students  to  work  on  individually. 

I have  had  short  informal  interviews  for  students  whose  problem-solving  skills  are  at  the 
preliminary  and  partial  response  stages.  (Hoping  to  organize  a more  formal  set  of  interviews  next 
semester.) 

I display  the  students’  solved  problems  each  week.  I now  have  materials  such  as  dice,  markers, 
squared  paper,  geometry  sets,  tape,  geometric  shapes,  etc.,  available  in  the  classroom,  rather 
than  retrieving  them  from  the  cupboards  on  demand. 

i now  mark  holistically  because  it  gives  a clearer  picture  of  the  strategies  students  are  using,  and 
I also  have  a better  idea  about  which  students  would  benefit  from  interviews. 
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RESULTS  OF  A SURVEY  OF  TEACHER  PARTICIPANTS  IN 
THE  EQI-MATH  PROJECT 
MAY  1992 


L PARTICIPANTS 

Twenty-four  classroom  teachers  were  involved  in  the  development  and 
pilot  testing  of  materials  for  the  EQI-Math  project.  Sixteen  of  these 
teachers  helped  to  develop  the  materials  during  the  1989-90  and  1990-91 
school  years.  Five  of  the  16  were  involved  for  only  one  year  as  a result  of 
transfers  to  different  schools  and  other  assignments.  Eight  teachers  pilot- 
tested  the  materials  during  the  1991-92  school  year. 

A survey  was  sent  to  all  24  teacher  participants.  Seventy-nine  percent  (19) 
were  returned  and  processed. 

n.  PURPOSE 

The  purposes  of  the  survey  were  to  determine  (a)  teacher  practice  in 
teaching  problem  solving,  and  (b)  the  impact  of  the  EQI-Math  project  on 
teaching  practice  as  reported  by  the  participating  teachers  themselves. 
Another  major  aspect  of  the  survey  was  to  determine  areas  of  the  project 
that  were  particularly  successful,  and  areas  that  should  receive  additional 
attention  in  the  future. 

III.  RESULTS  OF  THE  SURVEY 

A.  Teacher  practice  in  problem  solving 

Teachers  were  asked  to  indicate,  on  a scale  of:  none  - little  - some  - 
much  -undecided,  the  degree  to  which  they  made  use  of  15  teaching 
techniques  which  contribute  to  the  development  of  student  problem- 
solving skills.  Teachers  reported  that  all  but  one  were  using  "some”  or 
"much"  in  their  teaching  practice. 

The  exception  was:  Having  students  take  problems  home,  thereby 
inviting  parent  participation,  which  received  a rating  of  **little  - 
some,  ” The  others,  in  order  of  usage  from  "much  " to  "some, " were: 

Techniques  receiving  "much"  usage: 

- Ask  students  to  explain  how  they  reached  their  answers. 

- Have  students  working  in  pairs  or  groups. 

- Illustrate,  model  and  discuss  strategies. 
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- Talk  out  problems  indicating,  for  example,  what  I see,  what  I think, 
how  I would  proceed. 

- Encourage  students  to  acknowledge  methods  that  are  different 
from  the  ones  they  applied  to  a particular  problem. 

- Present  problems  that  have  more  than  one  solution. 

- Provide  students  with  a variety  of  problems  which  will  require 
utilization  of  different  strategies. 

- Utilize  brainstorming  techniques. 

- Have  students  present  solutions  to  the  class. 

- Have  students  demonstrate  to  others  or  to  the  class.  (This  received 
"much”  emphasis  in  Grades  5 and  8,  "some  emphasis  in  Grade  2,  and 
"least"  emphasis  in  Grade  10.) 

Techniques  receiving  **some**  usage: 

- Have  debriefing  sessions  in  which  students  can  talk  about  what 
was  successful  or  not  successful  in  their  problem-solving 
approaches. 

- Apply  a problem-solving  model  to  a real-world  problem  that  has 
nothing  to  do  with  Mathematics.  (Received  "much”  emphasis  in 
Grades  5 and  8,  "some"  emphasis  in  Grade  2,  and  "little"  emphasis  in 
Grade  10.) 

- Work  through  problems  with  individual  students. 

- Have  students  make  up  their  own  problems.  (This  received  "some" 
emphasis  at  the  elementary  level  and  between  "little"  to  "some"  at 
the  secondary  level.) 

Comments 

With  the  exception  of  the  qualifications  noted  in  three  of  the  items 
above,  there  was  considerable  consistency  in  the  utilization  of  the  15 
techniques  among  teachers  from  all  grade  levels. 

It  was  not  possible  to  collect  "before"  and  "after"  types  of  data  on 
teacher  utilization  of  these  problem-solving  techniques  as  a result  of 
participation  in  the  EQI  project.  However,  further  study  might  be  useful 
in  determining  whether  the  use  of  the  EQI-Math  assessment  materials 
encouraged  the  increased  use  of  particular  teaching  strategies. 
Additional  study  would  be  useful  in  examining  the  connection 
between  teaching  and  assessment  strategies  and  student  success  in 
problem  solving. 

B.  Impact  of  EQI-Math  on  participating  teachers 

- Teachers  were  asked  to  respond  to  the  following  statements  on  a 
seven-point  scale:  strongly  agree  - agree  - agree  somewhat  - 
undecided  -disagree  somewhat  - disagree  - strongly  disagree: 
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Participation  in  the  EQI  project  has  been  a valuable /useful  learning 
experience  for  me. 

Response: 

• All  respondents  reported  in  the  "agree"  categories:  68%  "strongly 
agree,"  26%  "agree,"  5%  "agree  somewhat." 


My  involvement  in  EQI  has  made  me  a better  teacher  of  mathematics. 
Response: 

• 95%  responded  in  the  "agree"  categories:  58%  "strongly  agree," 
26%  "agree,"  11%  "agree  somewhat."  5%  (one  respondent)  was 
"undecided." 


I believe  that  participation  in  the  EQI  project  was  the  type  of 
experience  that  would  be  of  value  to  teachers  of  subject  areas  other 
than  mathematics. 

Response: 

• 95%  responded  in  the  "agree"  categories:  63%  "strongly  agree," 

32%  "agree."  5%  (one  responded)  "disagreed." 


The  problem-solving  materials  developed  in  the  EQI  project  are: 

a)  useful  to  me 

Response: 

• 80%  responded  in  the  "agree"  categories:  37%  "strongly  agree," 
42%  "agree,"  5%  "agree  somewhat."  5%  were  undecided  and  11% 
did  not  respond  to  the  item. 


b)  well- organized  and  presented 
Response: 

• 84%  responded  in  the  "agree"  categories:  42%  "strongly  agree," 

37%  "agree,"  5%  "agree  somewhat."  5%  were  undecided  and  11% 
did  not  respond. 
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c)  easily  understood 
Response: 

• 85%  responded  in  the  "agree”  categories:  32%  "strongly  agree," 

53%  "agree."  5%  were  undecided  and  11%  did  not  respond. 


d)  appropriate  for  the  grade  level(s)  I teach 
Response: 

• 84%  responded  in  the  "agree"  categories:  37%  "strongly  agree," 

42%  "agree,"  5%  "agree  somewhat."  5%  were  undecid^  and  11% 
did  not  respond. 


Comments: 

Undecided  and  non-respondents  to  the  above  question  might  be  those 
who  did  not  see  the  finalized  materials  used  during  the  pilot.  Pilot 
materials  were  not  sent  to  any  of  the  teacher  researchers. 

• Teachers  were  asked: 

To  what  degree  has  your  participation  in  the  EQI  project  increased 
your  skills  and  confidence  in  the  following  areas  of  mathematics , and 
a list  of  7 activities  followed. 

Response: 

- Teachers  in  all  grades  reported  that  participation  in  the  project 
increased  their  skills  and  confidence  most  greatly  ("much")  in  the 
area  of  "assessment  of  student  growth  in  problem  solving." 

- All  of  the  other  areas  received  a rating  of  "some"  increase  in 
confidence  and  skills: 

• discussing  progress  in  mathematics  with  students 

• skills  in  teaching  problem  solving 

• knowledge  of  problem-solving  techniques 

• reporting  student  progress  in  mathematics 

• using  new  teaching  strategies  in  mathematics 

• discussing  children's  progress  with  parents 
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MATH  PROBLEMS  USED  IN  PILOTING 


EQI2 

Problem  #1 

Mary  Jane  saw  8 rabbits. 

How  many  ears  did  those  rabbits  have  in  all? 

Show  and  explain  all  your  work. 

Circle  one  of  the  following  words  you  would  use  to  describe  this  problem: 

Boring  Challenging  Confusing  Fun 

Circle  one  of  the  following  words  that  describes  how  you  feel  after  trying  to 
solve  this  problem: 

Discouraged  Satisfied  Frustrated  Happy 


EQI5 

Problem  #5 

David  and  Jason  are  visiting  Calgary. 

One  day  while  exploring  the  city  they  realize  they  are  lost. 

David  said  to  Jason,  ’’When  we  were  at  the  Calgary  Tower  we  were  8 blocks 
west  of  the  hotel."  Jason  remembered  that  after  they  left  the  tower  they  went 
south  4 blocks,  then  east  for  3 blocks,  north  for  2 more  blocks,  then  east  for  5 
blocks. 

How  far  away  from  the  hotel  are  the  boys  when  they  realize  they  are  lost? 
Show  and  explain  all  your  work. 

Circle  one  of  the  following  words  you  would  use  to  describe  this  problem: 

Boring  Challenging  Confusing  Fun 

Circle  one  of  the  following  words  that  describes  how  you  feel  after  trying  to 
solve  this  problem: 

Discouraged  Satisfied  Frustrated  Happy 
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EQIIO 


Problem  #4 

A. 

Take  any  four  consecutive  numbers;  multiply  them  together  and  add  1. 
Take  the  square  root  of  the  number. 

Is  your  final  answer  a whole  number? 

Show  and  explain  all  your  work. 

B. 

Take  the  middle  pair  of  numbers  and  multiply  them  together. 

Compare  with  your  first  answer. 

C. 

Repeat  parts  A and  B three  more  times  and  write  down  a possible  rule. 


Circle  one  of  the  following  words  you  would  use  to  describe  this  problem: 

Boring  Challenging  Confusing  Fun 

Circle  one  of  the  following  words  that  describes  how  you  feel  after  trying  to 
solve  this  problem: 

Discouraged  Satisfied  Frustrated  Happy 
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APPENDIX  D 


EDUCATIONAL  QUALITY  INDICATORS  PROJECT: 


EVALUATION  OF  PROJECT  UTILITY, 


MANAGEMENT  & IMPACT 


July  29,  1992 
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1 .0  BACKGROUND  TO  THE  EVALUATION  OF  PROJECT  MANAGEMENT 


The  Calgary  Educational  Quality  Indicators  (EQI)  Project  was  a three-year  action  research 
project  (1989  - 1992)  funded  by  Alberta  Education  to  develop  indicators  of  quality  student 
performance  in  school  art  composition  and  mathematics  problem  solving  at  the  Calgary 
Board  of  Education  and  the  Calgary  Catholic  Board  of  Education. 

The  project  purposes  were  defined  as  follows: 

1.  To  identify  indicators  of  quality  work  which  can  be  applied  in  the  assessment 
of  student  achievement  in  areas  of  Art  and  Mathematics 

2.  To  identify  conditions  that  appear  to  enhance  or  limit  the  achievement  of 
quality 

3.  To  develop  strategies  for  collecting  quality  student  work  which  include 
recorded  interviews  with  teachers  and  students 

4.  To  examine  quality  work  with  emphasis  on  alternative  assessment  strategies 
to  paper  and  pencil  testing 

The  project  was  part  of  a larger,  province-wide  initiative  sponsored  by  Alberta  Education 
to  build  sets  or  clusters  of  indicators  to  use  in  a broader  approach  to  assessment.  A number 
of  other  jurisdictions  were  involved  as  well,  looking  at  a variety  of  assessment  topics. 

The  Calgary  project  focused  on  assessing  performance  in  the  areas  of  art  and  mathematics. 
These  subject  areas  were  chosen  because  of  preliminary  work  which  had  already  been  done 
at  the  Calgary  Catholic  School  Board  in  the  area  of  art  assessment  and  because  of 
mathematics  expertise  at  the  Calgary  Board  of  Education.  Further,  it  was  reasoned  that  the 
two  areas  could  allow  a unique  exploration  of  the  assessment  of  product  and  process.  The 
art  project  could  examine  the  product  of  the  learning,  as  much  of  art  education  had  stressed 
process  in  the  past,  and  the  math  project  could  examine  the  process  of  student  thinking,  as 
most  math  education  had  emphasized  the  product,  or  answer,  in  the  past. 

The  project  design  incorporated  three  phases:  development,  field  testing  and  pilot  testing 
of  quality  indicators.  Committees  of  teachers  and  subject  specialists  reviewed  student  work, 
identified  characteristics  of  quality  work,  considered  alternative  approaches  for  recording 
assessment  information  and  proposed  activities  and  materials  for  use  in  classrooms. 
Classroom  teachers  tried  out  a variety  of  materials  and  approaches  with  their  students, 
materials  were  revised  and  then  were  piloted  with  other  teachers. 

The  project  had  a full-time  designated  director  who  was  located  at  the  Calgary  Board  of 
Education.  A steering  committee,  comprised  of  school  superintendents  and  representatives 
from  Alberta  Education  provided  coordination  and  leadership.  A community  advisory 
committee,  comprised  of  university  professors,  leaders  in  the  fields  of  art  and  mathematics, 
parent  representatives,  representatives  of  the  Alberta  Teachers’  Association,  and  members 
of  the  business  community  provided  a sounding  board  to  test  project  ideas.  Finally,  two 


1 


subject  area  advisory  committees,  with  specialists  from  both  boards,  provided  "hands  on" 
decision  making  for  the  action  research  in  each  area. 

The  art  component  of  the  project  culminated  in  the  development  of  the  Educational  Quality 
Indicators  Handbook  which  included  colour  reproductions  of  quality  samples  of  students’ 
art  from  Grades  2 and  5,  and  from  junior  and  senior  high  along  with  teacher  expectations 
for  each  lesson  and  student  comments  assessing  their  own  work.  Three  quality  indicators 
emerged  from  the  project  as  observable  characteristics  to  be  found  in  students’  work, 
including  relationshipsy  handling  and  meaning.  Student  and  teacher  journals  were  used  in  the 
pilot  test  to  assist  in  reflective  practice  on  the  part  of  both  teachers  and  students. 

Project  conclusions  for  the  art  component  included  the  following: 

1.  Quality  assessment  practice  supports  improved  art  making. 

2.  Attending  to  written  and/or  spoken  student  responses,  reflections  and 
thoughts  illuminates  the  art  work. 

3.  Quality  art  assessment  is  a shared  experience  between  students  and  teachers. 

4.  Effective  art  assessment  depends  on  common  vocabulary  shared  by  teachers 
and  students. 

5.  The  collection  of  quality  art  exemplars  emphasizes  diversity  as  an  attribute  of 
a quality  art  program. 

6.  Expert  professional  judgement  benefits  from  collegial  work. 

7.  Assessment  takes  time. 

8.  EQI  Handbook  exemplars  are  a way  to  share  expertise,  to  develop 
coimoisseurship. 

Tlie  math  component  explored  the  concept  of  holistic  scoring  and  developed  some  response 
criteria  including  preliminary  responsej  partial  response,  complete  response  and  elegant 
response.  A series  of  students’  mathematical  dispositions  were  identified,  including 
motivated,  creative,  confident,  strategic  approach  and  strategic  process.  Conditions  for  the 
math  learning  environment  were  also  considered. 

Project  conclusions  for  the  mathematics  component  included  the  following: 

1.  Teachers  report  positive  changes  to  their  teaching  and  assessment  practices 
as  a result  of  participating  in  EQI-Math. 

2.  Teachers  want  to  meet  together  to  discuss  teaching  and  assessment.  The 
experience  of  working  closely  with  colleagues  is  very  positively  received. 
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3.  Useful  materials  and  practices  developed  in  the  project  included  student 
dispositions,  holistic  scoring  and  the  exploration  of  Averse  student  responses 
and  quality  solutions  to  mathematical  problems. 

4.  The  use  of  anecdotal  records,  student  attitude  reports  and  generalizations 
about  classroom  conditions  was  not  seen  as  praaic^,  due  to  time  and  energy 
constraints. 

5.  Students  seemed  to  be  more  positive  about  problem  solving,  took  more  risks, 
and  demonstrated  more  confidence  as  a result  of  the  EQI  project. 

6.  Systematic  methods  for  observing  students  closely  helped  teachers  with 
teaching  decisions  and  with  assessment  and  evaluation  tasks. 

7.  Students  respond  to  the  invitation  to  express  themselves  more  fully  and  to 
think  about  their  thinking. 

Finally,  some  commonalities  emerged  about  the  making  of  art  and  the  solving  of 
mathematical  problems  in  terms  of  the  complex  thinking  processes  required  for  each  and 
also  in  terms  of  the  assessment  challenges  they  present  teachers.  Common  fmdings  included 
the  following: 


1.  Quality  assessment  practice  in  art  and  mathematics  supports  teacher  decision 
making  about  teaching  and  assessment. 

2.  Attending  to  written  and  spoken  student  responses,  reflections  and  thoughts 
illuminates  the  art  work  or  the  problem  solution. 

3.  Quality  assessment  is  a shared  experience  between  teachers  and  students 
which  is  encouraged  by  shared  exemplars  and  a shared  vocabulary. 

4.  Qualitative  assessment  takes  time  for  both  teachers  and  students. 

5.  Exemplars  are  a way  to  share  expertise,  develop  connoisseurship,  establish 
common  expectations  for  student  outcomes  and  celebrate  diversity  of  student 
response. 

The  evaluation  of  project  management  at  the  local  level  was  mandated  to  be  an 
independent,  third-party  evaluation  and  as  such  was  contracted  out  to  an  independent 
program  evaluator. 

A series  of  research  questions  was  developed  in  collaboration  with  the  Project  Director. 
They  were  as  follows: 

1.0  UTILITY 

1.1  How  useful  are  the  EQI  elements  to  assess  student  achievement? 

1.2  Can  they  be  used  for  reporting  to  parents? 
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1.3  Can  EQI  materials  be  applied  to  other  subject  areas?  If  so,  what 
areas? 

1.4  How  would  you  model  or  depict  key  elements? 


2.0  PROJECT  MANAGEMENT 

2.1  Were  the  necessary  management  systems  in  place  to  support  successful 
project  implementation? 

2.2  What  impact  did  the  steering  committee  have  on  project 

implementation? 

23  What  impact  did  the  advisory  committee  have  on  project 

implementation? 

2.4  Did  the  partnership  of  the  two  school  systems  affect  either  project 
implementation  or  outcomes?  If  so,  how? 


3.0  PROJECT  IMPACTS/RECOMMENDATIONS 

3.1  What  recommendations  would  you  make  to  improve  project  utility? 

3.2  What  recommendations  would  you  make  to  improve  project 
management? 

33  What  happened  as  a result  of  the  EQI  project  that  you  did  not  expect? 

3.4  What  impacts  will  the  EQI  project  have  on  education  in  Alberta? 

A series  of  eleven  face-to-face  interviews  were  held  with  representatives  of  the  two  school 
jurisdictions  and  Alberta  Education.  A breakdown  is  as  follows: 


Calgary  Board  of  Education  Senior  Administration  1 

Project  Director  1 

Math  Education  2 

Art  Education  2 

Calgary  Catholic  School  Board  . Senior  Administration  1 

Math  Education  1 

Art  Education  2 

Alberta  Education  Senior  Manager  1 


The  data  were  compiled  and  analyzed  for  content  after  Krippendorff  (1980).  The  remainder 
of  this  report  is  a summary  of  responses  for  each  of  the  research  questions  outlined  above 
followed  by  some  conclusions. 
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2.0  RESEARCH  FINDINGS 


Each  of  the  research  questions  will  be  explored  in  turn.  The  comments  were  grouped  and 
analyzed  according  respondents*  areas  of  expertise  (art,  mathematics  or  supervision),  and 
according  to  employer  (Calgary  Board  of  Education,  Calgary  Catholic  School  Board  or 
Alberta  Education)  to  search  for  similarities  or  differences.  Where  appropriate,  these 
similarities  or  differences  will  be  identified. 


1.0  UTILITY 


1.1  How  useful  are  the  £QI  elements  to  assess  student  achievement? 

Generally,  respondents  found  the  art  EQI  elements  to  be  very  useful.  The  art  specialists 
agreed  that  both  the  visuals  in  the  handbook  and  the  concepts  of  meaning,  relationships  and 
handling  would  be  valuable  to  both  the  specialist  and  the  generalist.  The  fact  that  the 
student  was  present  for  the  assessment  process  and  able  to  talk  in  a shared  language  about 
the  artwork  was  seen  to  be  very  positive  as  insight  could  thus  be  gained  into  how  to  make 
that  artwork  stronger.  The  distinction  between  assessment  and  evaluation  was  seen  as  a 
useful  one.  However,  it  was  pointed  out  that  some  orientation  would  be  required  for  both 
teachers  and  students  to  interact  successfully  with  the  handbook. 

The  math  specialists  were  divided  in  their  views  about  how  useful  the  math  EQI  elements 
would  be  to  assess  student  achievement.  On  the  positive  side,  it  was  seen  that  the  elements 
provided  some  useful  language  to  describe  what  teachers  should  be  looking  for  in  problem 
solutions  and  what  dispositions  students  should  exhibit.  As  one  specialist  commented: 

The  holistic  approach  makes  us  look  at  the  solutions  to  problems  and  not  just  at  the 
answers. 

Another  aspect  of  the  project  which  was  considered  useful  was  the  opportunity  it  provided 
for  teacher  collaboration. 

However,  the  progress  made  in  the  math  component  was  seen  as  "a  first  small  step”, 
consistent  with  current  thinking  in  math  research,  but  the  project  did  not  provide  any 
conclusive  links  between  student  dispositions  and  exemplary  work.  As  one  individual 
commented: 

This  is  not  the  definitive  answer,  it  is  some  information. 

Two  of  the  four  supervisors  interviewed  were  positive  in  their  responses,  the  others  were 
more  qualified.  The  elements  were  seen  to  be  useful  because  they  reflected  emerging 
educational  thought  about  reflective  teaching,  self-assessment,  authentic  assessment  and 
performance  assessment.  As  one  commented: 

There  are  new  ways  of  seeing  new  words  to  use  and  new  ways  to  gather  information. 
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However,  it  was  also  felt  that  the  materials  would  have  to  gain  respect  on  their  own  merit. 
Teachers  were  inundated  with  materials  and  would  have  to  see  the  handbook  as  instantly 
useful  and  applicable  if  they  were  to  explore  it  further.  It  was  the  belief  of  one  supervisor 
that  math  was  a higher  priority  than  art  for  elementary  teachers  and  therefore  the  art 
handbook  was  less4ikely  to  be  adopted  by  them,  whereas  art  teachers  in  high  school  were 
specialists  and  might  be  more  likely  to  use  the  elements.  Another  supervisor  felt  that  the 
art  handbook  would  be  used  but  that  as  the  outcomes  in  math  were  less  tangible,  they  might 
be  less  likely  to  be  utilized  as  a result.  The  fear  was  expressed  that  the  issue  of 
institutionalizing  project  concepts  had  yet  to  be  addressed  and  that  the  lack  of  involvement 
on  the  part  of  senior  administration  in  both  boards  was  a deterrent  to  implementation. 

Overall,  with  the  exception  of  the  art  supervisors  from  both  boards  who  were  positive  in 
their  responses,  respondents  from  the  Calgary  Board  of  Education  viewed  the  project 
elements  as  being  more  useful  than  those  from  the  Calgary  Catholic  School  Board. 


12  Can  they  be  used  for  reporting  to  parents? 

The  art  specialists  were  positive  in  their  views  that  exemplars  of  students’  art  in  the 
handbook  would  be  very  useful  in  talking  to  parents  as  it  was  seen  as  a good  place  to  begin 
a dialogue  about  their  children’s  work.  In  fact,  pilot  teachers  indicated  that  this  was  the 
case.  One  teacher  had  reported  a conversation  with  a parent  as  follows: 

...  he  realized  for  the  first  time  how  powerful  art  is  in  a child's  learning. 

It  was  felt  that  the  EQI  handbook  would  be  used  as  a model  for  anecdotal  reporting  in  the 
future. 

The  math  specialists  were  divided  in  their  views  about  using  EQI  elements  for  reporting  to 
parents.  One  felt  that  a lot  more  understanding  had  to  be  developed  about  how  children 
learn  Math  before  performance  assessment  would  be  an  acceptable  reporting  format  or 
teachers’  credibility  would  be  at  risk.  In  addition,  this  individual  felt  that  parents  were  not 
yet  ready  to  deal  with  the  qualitative  nature  of  this  approach.  The  other,  however,  quoted 
a pilot  teacher  who  had  said: 

When  I did  my  report  cardSy  I knew  my  kids  better  this  time. 

This  individual  felt  that  looking  at  dispositions  and  extended  solutions  was  helpful  in 
reporting  to  parents. 

Generally  the  supervisors  were  positive  in  their  view  that  EQI  elements  could  be  used  for 
reporting  purposes.  They  saw  both  parents  and  the  business  community  as  being  open  to 
new  forms  of  assessment.  However,  probably  due  to  their  administrative  perspective,  they 
had  several  suggestions  about  what  would  have  to  happen  first  These  included: 

1.  The  need  for  parent  partnership  in  their  child’s  assessment 

2.  The  need  for  guidelines  for  performance  assessment  available 
to  the  public 


6 


3. 


The  need  for  public  and  parent  education 


They  saw  a number  of  advantages  to  using  alternative  assessment  tools,  including  parent 
ownership  in  the  child’s  assessment,  a shared  understanding  of  quality  in  the  community  and 
a means  of  acknowledging  risk  taking  and  alternative  solutions,  not  just  correction  and 
control.  One  supervisor  did  feel,  however,  that  reporting  would  have  to  take  a different 
format  from  those  developed  in  the  projea  before  it  would  be  ready  for  the  public  but 
another  hoped  that  the  elements  would  be  used  for  reporting  and  commented: 

Far  more  parents  want  this  land  of  evaluation  than  do  those  who  want  a letter 
grade.  When  I show  them  the  richness  of  information  from  the  kids'  ''talk"  they 
are  positive.  With  thoughtful  discussion  with  parent  groups^  it  has  great  promise. 

An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards. 


13  Can  £QI  materials  be  applied  to  other  subject  areas?  If  so,  which 
areas? 

This  question  elicited  a number  of  comments  on  areas  of  transfer.  Every  respondent  felt 
that  there  were  lessons  to  be  learned  from  the  EQI  project  and  that  either  parts  or  all  could 
be  transposed  to  other  subject  areas. 

Two  of  the  math  supervisors  felt  that  both  the  quality  indicators  and  the  project  process 
could  be  applied  to  any  subject.  As  one  commented: 

Teachers  of  all  subject  areas  need  to  identify  problems  for  students,  get  responses,  and 
sit  down  as  educators  to  look,  read,  analyze  and  discuss  student  responses  and  to  set 
benchmarks  about  realistic  expectations  for  certain  ages  of  children.  The  process  to  find 
out  about  what  teachers  think  about  responses  can  be  done  in  any  area. 

In  addition,  the  interaction  which  had  occurred  between  the  art  and  math  projects  was  seen 
as  an  example  of  transferability: 

They  have  words  to  describe  the  art  and  they  can  now  be  applied  to  math.  Math  could 
take  the  reflection  component  from  art.  It  would  be  the  next  step  to  have  kids  re  flea  on 
their  solution  and  give  them  the  words  to  talk  about  solutions  (self-assessment).  You 
could  take  it  anywhere.  It  is  valuable  to  assess  the  process. 

The  art  specialists  saw  a particularly  close  link  between  art,  writing  and  social  studies.  A 
similarity  was  seen  between  the  way  art  was  edited  and  critiqued  in  the  EQI  project  and  the 
way  students’  writing  can  be  assessed  and  edited  so  that  students  take  ownership  of  the 
process.  The  use  of  positive  exemplars  was  seen  as  another  transferrable  instructional 
method  as  was  the  use  of  journals  to  provide  the  opportunity  to  capture  reflection.  As  one 
individual  commented: 

Children  are  looking  for  meaning. . . . The  insight  into  students'  reflection  on  their  work 
is  very  rich. 


1 


Social  studies  was  seen  to  have  a strong  link  to  historical  and  visual  materials  so  that  the 
EQI  concepts  of  meaning,  relationships  and  handling  were  seen  to  be  applicable  to  culture 
and  environment  as  well. 

The  supervisors  tended  to  agree  that  the  EQI  concepts  were  transferrable.  In  addition,  the 
view  was  advanced  that  they  related  to  inter-subject  areas  as  well  such  as  English  and  social 
studies  where  the  use  of  portfolio  assessment  was  already  in  place: 

It  fits  to  have  them  look  at  dispositions,  as  the  math  dispositions  are  really  learning 
dispositions.  Art  has  developed  generic  criteria.  The  invitation  to  let  students  assess 
themselves  and  to  develop  criteria  about  what  is  successful,  to  internalize  quality 
standards,  show  samples,  use  vocabulary  to  look  at  work  in  progress  [all  fit]. 

It  was  pointed  out  that  there  already  were  writing  standards  developed  and  that  different 
subjects,  such  as  industrial  arts,  already  used  samples  of  finished  products  as  benchmarks. 
In  addition,  teachers  used  holistic  indicators  to  assess  project  work  (e.g.,  a puppet  play  for 
a book  report  assignment).  A close  link  was  seen  between  math  and  science  in  terms  of 
problem  solving  skills. 

To  summarize,  project  concepts  were  seen  to  be  transferrable  to  the  following  subjects  (with 
frequency  of  comment  presented  in  parentheses): 


All  subjects 

(2) 

Language  arts 

(2) 

Science 

(2) 

Social  studies 

(1) 

Inter-subjects 

(1) 

However,  two  respondents  had  reservations.  One  felt  that  the  project  had  developed 
generic  impressions  but  that  each  subject  area  would  have  to  develop  specific  descriptors. 
However,  this  individual  did  believe  that  the  dispositions  and  response  criteria  developed 
in  the  math  component  had  some  applicability  to  other  areas. 

The  other  respondent  believed  that  a prototype  had  not  yet  been  pulled  out  of  the  project 
that  could  be  generalizable.  While  the  project  had  been  an  excellent  experience  for  those 
involved  in  it,  the  EQI  findings  could  not  in  their  current  form  find  widespread  applicability. 

An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards. 


1.4  How  would  you  model  or  depict  key  elements? 

This  question  was  difficult  for  respondents  to  answer.  No  one  came  up  with  a definitive 
model  for  the  project  However  a number  of  components  were  identified  which  could  be 
massaged  into  a model  which  would  depict  the  following: 
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Student-Teacher  Interaction 


Students  and  teachers  involved  in  a meaningful, 
equal,  reflective,  critical  activity  using  shared 
language  in  order  to  problem  solve  and  add 
further  understanding 

While  no  one  had  a complete  model,  some  interesting  components  were  identified  which 
could  be  extraaed  for  further  analysis.  These  included  the  following: 

♦ Inter-teacher  dialogue,  analysis  and  collaboration  regarding  student  responses 

♦ Samples  of  exemplary  work  to  develop  a shared  understanding  of  quality 

♦ Exploration  of  communications  about  problem  solving 

♦ Use  of  journals  to  explore  student  reflection  to  access  the  richness  of  student 
thought  (mirrors  of  the  mind) 

♦ The  paucity  of  information  provided  by  traditional  paper  and  pencil  testing 

♦ The  relationships  between  the  amount  of  space  provided  to  capture  student 
responses  on  paper  and  the  quantity  of  reflective  comments  elicited 

Staff  members  from  the  Calgary  Board  of  Education  seemed  to  have  more  suggestions 
regarding  a potential  model  than  other  respondents  did. 


2.0  PROJECT  MANAGEMENT 

2.1  Were  the  necessary  management  systems  in  place  to  support  successful 

project  implementation? 

A number  of  positive  comments  were  elicited  regarding  the  management  and  management 
systems  for  the  implementation  of  the  EQI  project.  These  included  the  following  (with 
frequency  of  response  in  parentheses): 

Quality  of  Project  Direction  (4^ 

The  Project  Director  was  cited  for  her  excellent  job,  sometimes  in  trying 
circumstances,  in  terms  of  background  reading  and  preparation,  keeping 
everyone  on  task,  writing  and  editing,  and  external  communications.  The 
project  was  seen  to  have  run  smoothly  largely  due  to  her  direction. 

Adequate  Budget  (2  pro.  1 con) 

The  budget  was  seen  to  be  adequate  to  run  the  project  by  two  respondents 
but  the  third  described  it  as  "bare  bones".  However,  the  project  was  never 
hindered  from  lack  of  money  and  was  even  able  to  produce  the  art  Handbook 
with  costly  reproductions. 
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Adequate  Time  (2) 


The  three  years  were  seen  to  be  adequate  for  project  development.  There 
were  comments,  however,  that  one  more  year  would  have  been  helpful  to 
disseRiinate  and  institutionalize  the  concepts. 

Facilities  and  Secretarial  Support  (1  each) 

The  location  of  the  project  in  the  Calgary  Board  of  Education’s  head  office 
was  seen  as  helpful  in  that  adequate  space  and  secretarial  support  were 
provided.  However,  it  was  the  view  of  the  project  direaor  that  she  might 
better  have  been  located  on  site  with  one  of  the  project  components. 

Networking 

The  housing  of  the  project  director  in  the  Calgary  Board  of  Education’s 
Department  of  Curriculum  allowed  her  to  be  hooked  into  a variety  of 
curriculum  networks  at  both  the  local  and  provincial  level  which  facilitated 
the  flow  of  information  between  the  projea  and  its  larger  environment. 


On  the  other  hand,  a number  of  problems  related  to  project  implementation  were  also  cited. 
These  included  the  following: 


Turnover  in  Project  Representation  in  the  Math  Area  (4^1 

While  the  art  representation  and  leadership  remained  constant  and  strong 
throughout  the  project,  the  math  component  suffered  from  the  turnover  of 
several  members,  including  the  curriculum  leader  from  the  Calgary  Board  of 
Education  and  the  specialists  from  the  Calgary  Catholic  School  Board.  It  was 
felt  that  the  turnover  slowed  the  process,  and  led  to  a lack  of  clarity  regarding 
project  goals  and  boundaries.  It  was  felt  that  the  math  team  never  "clicked" 
and  had  less  commitment  to  the  project  which  had  a negative  impact  on 
project  outcomes. 


Lack  of  Involvement  of  Board  Superintendents 

Respondents  commented  that  the  superintendents  of  both  boards  did  not  take 
ownership  for  the  project.  Despite  their  involvement  in  proposal  development 
and  the  acquisition  of  funding  for  the  project  initially,  they  were  only 
minimally  involved  at  the  Steering  Committee  level  (see  below)  and  did  not 
make  the  project  a high  priority.  The  project  director  was  seen  to  be  on  her 
own  with  an  orphan  project  to  run.  This  was  contrasted  with  other  provincial 
projects  where  district  superintendents  demonstrated  ownership  and 
involvement.  It  was  feared  that  this  lack  of  engagement  and  buy-in  would 
have  an  impact  in  terms  of  the  institutionalization  of  project  concepts. 
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Lack  of  Involvement  of  Project  Initiators  in  Project  Implementation  (2) 


The  initial  proposal  was  written  by  central  office  personnel  from  the  two 
school  boards,  including  a member  of  the  Research  and  Testing  Department 
of  the  Calgary  Board  of  Education  (now  the  Assessment  Team),  with 
consultation  from  one  art  specialist.  The  proposal  writers  never  worked  on 
the  project  and  projea  advisors  had  trouble  clarifying  project  goals.  Time  was 
lost  interpreting  what  they  had  in  mind  and  was  particularly  problematic  for 
the  math  component 


Prpjea.,Dire<PtQr.  PpsitiQ.nXU 

The  fact  that  the  project  director  held  a term-specific  assigmnent  which 
terminates  at  the  end  of  the  EQI  project  rather  than  holding  a line  position 
in  either  board  was  also  seen  to  be  a deterrent  to  potential  future 
institutionalization  of  project  concepts. 


An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards. 


22  What  impact  did  the  steering  committee  have  on  project 
implementation? 

A project  steering  committee  was  comprised  of  the  two  board  superintendents,  the  subject 
area  leaders,  representatives  from  Alberta  Education,  and  the  project  director.  The  purpose 
of  this  committee  was  to  monitor  project  developments  and  receive  and  approve  project 
reports.  Two  meetings  were  held  during  the  initial  phase  of  the  three-year  project,  but  then 
the  function  of  this  committee  was  incorporated  into  existing  advisory  committees.  The 
Alberta  Education  representative  helped  connect  the  project  with  the  provincial  EQI 
director  for  decision-making. 

Identification  of  membership  on  the  steering  committee  was  limited.  The  various  steering 
and  advisory  committees  were  confused  in  most  respondents’  minds  in  terms  of  both 
membership  and  function.  The  project  director’s  contract  stated  that  she  was  to  report  to 
the  steering  committee  regularly  and  to  receive  feedback  and  advice  from  the  committee. 
Interim  project  reports  were  sent  to  the  provincial  co-ordinator  as  part  of  the  project 
contract  and  feedback  was  received. 

One  issue  which  was  discussed  at  the  steering  committee  level  was  that  of  qualitative  versus 
quantitative  analysis  of  study  outcomes.  The  provincial  EQI  initiative  advocated  a balanced 
approach  to  measuring  outcomes.  The  Calgary  project  focused  more  on  qualitative  methods 
and,  therefore,  some  quantative  data  which  were  collected  during  the  project  may  not  have 
been  included  in  the  final  report 
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An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinion  overall  between  the 
two  boards. 


23  What  impact  did  the  advisory  committees  have  on  project 
implementation? 

There  were  three  advisory  committees.  One  had  community  representation,  and  the  other 
two  were  working  committees  in  the  subject  areas.  As  a result,  responses  varied  depending 
on  the  perspective  of  the  respondent 

The  art  advisory  committee  was  seen  as  having  a major  impact  on  the  project.  The  project 
made  decisions  about  the  project,  guided  implementation,  debriefed  after  project  activities, 
and  interpreted  results.  It  was  from  this  level  that  the  action  research  emanated  and  the 
process  was  seen  to  work  very  well: 

77ie  teachers  found  it  to  be  a rewarding  professional  experience  to  sit  with  experts  and 
talk  about  teaching  art,  methodologies  and  philosophies.  They  referred  to  it  as  a 
collegial  model  and  recommended  it  as  a PD  model  They  then  used  the  same  model 
with  the  child.  The  answers  were  not  there  at  the  beginning.  Teachers  became  inventors 
and  researchers  and  became  part  of  the  process. 

In  addition,  it  was  suggested  at  the  Art  advisory  committee  that  journal  writing  be 
incorporated  into  the  project.  This  turned  out  to  be  one  of  the  positive  results  of  the 
project. 

The  math  advisory  committee  was  also  seen  to  have  a major  impact  on  project  direction, 
although  this  was  mitigated  to  a degree  by  lack  of  continuity.  The  committee  was  seen  as 
a positive  force  in  the  project  and  committee  members  were  involved  with  teacher  in- 
services and  classroom  implementation.  It  met  frequently  to  react  to  proposed  math 
problems  and  student  responses.  As  one  member  of  the  team  commented: 

We  kept  the  project  going  in  a current  direction  in  tune  with  the  literature  in  math 
education.  I don't  think  teachers  would  have  done  that  on  their  own  We  were  able  to 
stand  back  and  take  a look  at  new  directions. 

The  third  community  advisory  committee  was  seen  by  the  project  director  as  a good 
sounding  board  to  try  out  project  ideas  on  a wider  audience. 

It  appears,  overall,  that  the  two  subject  area  advisory  committees  were  the  heart  of  the 
project,  in  conjunction  with  the  classroom  teachers  and  students  who  implemented  their 
ideas. 

An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards. 
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2.4  Did  the  partnership  of  the  two  school  ^sterns  affect  either  project 
implementation  or  outcomes?  If  so,  how? 

Respondents  were  positive  in  their  views  about  the  impact  the  partnership  between  the  two 
boards  had  on  the ‘project.  Positive  outcomes  included  the  following  (with  frequency  of 
responses  in  parentheses): 


It  provided  an  opportunity  for  new  linkages  and  networking  (^6^ 

It  was  seen  as  particularly  valuable  that  teachers  from  the  two  boards,  who 
did  not  know  each  other  prior  to  the  project  and  who  had  few  opportunities 
in  the  past  to  work  together,  could  come  together  for  profession^  dialogue. 


It  was  a good  example  of  a partnership  (6"! 

The  project  gave  the  two  boards  a reason  to  communicate  in  a positive  way 
in  depth  about  a professional  topic.  This  was  seen  as  a unique  experience. 


It  was  a positive  professional  development  experience  (2) 

Project  involvement  resulted  in  an  increase  in  the  overall  base  of  expertise 
among  project  teachers  and  advisors. 


It  allowed  two  different  systems  to  work  together  (D 

The  two  school  systems  were  seen  to  have  very  different  organizational 
structures  and  management  styles.  The  cross-fertilization  of  ideas  was  seen 
as  a positive  outcome. 

There  were  few  negative  outcomes  identified.  It  was  mentioned  that  as  the  project  director 
was  housed  at  the  Calgaiy  Board  of  Education,  it  necessarily  followed  that  the  bulk  of 
project  costs  were  supported  by  this  board.  However,  it  was  also  noted  that  staff  from  both 
boards  cdntributed  equally  to  the  project,  equal  numbers  of  students  were  involved,  and  an 
equal  amount  of  ownership  was  demonstrated. 

An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards.  Both  were  equally  positive  about  the  experience. 
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3.0  PROJECT  IMPACTS/RECOMMENDATIONS 


3.1  What  recommendations  would  you  make  to  improve  project  utility? 

The  art  specialists  had  few  suggestions  for  improvement  of  the  project  and  generally  felt 
very  pleased  with  the  outcomes.  It  was  suggested,  however,  that  the  art  handbook  binder 
format  could  be  improved  upon  and  the  suggestion  was  made  that  an  accompanying  video 
of  teacher-student  interactions  would  be  helpful  as  well. 

The  math  specialists  reflected  on  the  fact  that  their  component  had  started  at  a less 
developed  level  of  conceptualization  than  the  art  component  and  as  a result  had  farther  to 
go: 


They  were  looking  at  the  product.  We  had  to  look  at  the  process  and  had  never  done 
much  in  that  area.  The  product  in  math  is  a lot  simpler.  In  art  there  is  no  right  answer 
so  they  had  explored  process  already.  We  got  teachers  focusing  on  the  solution  as  a 
documentation  of  the  thought  process.  To  improve  project  utility  would  be  to  have  kids 
talk  or  write  about  their  solutions.  Documentation  of  Jdds'  reflections  on  their  solutions 
would  be  the  next  step  for  us. 

In  fact  this  individual  suggested  that  another  project  was  required  in  math  to  allow 
continued  pursuit  of  these  preliminary  findings. 

Another  respondent  asked  what  would  have  happened  if  the  students  had  generated  the 
math  questions  rather  than  the  teachers,  and  further,  whether  or  not  the  teacher-generated 
questions  were  the  right  ones  for  each  grade  level.  This  individual  felt  that  a lot  more 
exploration  was  required  in  this  area. 

Another  considered  the  frustration  experienced  in  participating  in  this  developmental  project 
in  contrast  to  the  typical  mathematics  educator’s  experience  which  focused  on  the  bottom 
line.  The  amount  of  time  invested  in  the  project  with  little  demonstrable  output  was  a 
concern. 

The  supervisors  had  a number  of  suggestions  to  improve  project  utility  from  their 
perspective.  These  are  summarized  below. 


Ensure  implementation  at  the  Calgarv  Board  of  Education 

Due  to  both  differing  structures  within  the  two  boards  and  to  recent  financial 
decisions  made  at  the  Calgary  Board  of  Education,  it  was  felt  that  the 
dissemination  of  project  concepts  was  threatened  at  the  CBE  because  of  lack 
of  personnel  to  spearhead  the  process.  As  the  Assessment  Team  had  not 
been  involved  in  the  project,  it  was  not  anticipated  that  this  area  would 
continue  the  development  of  the  project  in  the  future.  The  subject  area 
specialists  were  being  reassigned  due  to  budget  constraints  and  so  would  not 
be  available  to  ensure  that  project  concepts  were  utilized.  The  project 
director’s  tenure  would  be  up  at  the  end  of  the  1991  - 1992  school  year  and 
so  no  further  direct  leadership  was  envisioned.  It  is  anticipated  that 
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curriculum  personnel  at  the  Calgary  Board  of  Education  will  utilize  the 
findings  of  the  project  as  they  work  in  their  inter-disciplinary  team 
assignments  in  schools. 

On  the  other  hand,  the  Calgary  Catholic  School  Board  was  retaining  their 
subject  area  specialist  structure,  and  four  of  the  sixteen  supervisors  had 
express  instructions  to  integrate  projea  concepts  and  practices  into  their  work. 

It  was  hoped  that  some  sort  of  partnership  arrangement  might  continue  so 
that  CBE  teachers  could  access  in-service  activities  in  this  area  at  the  Catholic 
board. 


Ensure  that  support  be  provided  on  project  concepts  (4^ 

It  was  felt  that  very  carefully  designed  teacher  support  would  be  required  to 
help  teachers  work  with  EQI  concepts.  It  would  be  important  for  discussion 
to  be  encouraged  among  groups  of  teachers,  that  mentoring  programs  be  set 
up  to  allow  for  on-going  dialogues  among  pilot  and  other  teachers  and  that 
principals  be  provided  with  learning  opportunities  so  that  they  could  support 
teachers  in  this  process.  The  outcomes  of  this  project  were  seen  as  a possible 
basis  for  a revision  of  student  assessment  but  the  process  would  have  to  be 
nurtured.  It  was  also  suggested  that  parents  would  require  background 
information  and  that  a parents’  primer  be  developed  for  use  at  Parents’ 
Advisory  Councils. 

Overall,  employees  of  the  Calgary  Board  of  Education  had  more  recommendations  to 
improve  project  utility  than  did  employees  of  the  Calgary  Catholic  School  Board.  Not 
surprisingly,  the  representative  from  Alberta  Education  strongly  wished  to  see  project 
concepts  built  into  practice  in  the  district. 


32  What  recommendations  would  you  make  to  improve  project 
management? 

Responses  to  this  question  tended  to  reflect  comments  made  earlier  in  the  interview  process. 
Two  major  recommendations  were  advanced  regarding  how  project  management  could  have 
been  improved,  as  follows: 


Involvement  of  senior  level  stakeholders  (3^ 

Lack  of  on-going  involvement  of  senior  management  at  both  boards  and 
Alberta  Education  was  seen  as  a drawback  to  project  implementation.  As  a 
general  thrust  toward  performance  assessment  is  perceived  in  educational 
circles,  it  was  felt  that  this  project  should  have  received  more  attention  than 
it  did. 
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Involvement  of  project  initiators  (2^ 


While  several  of  the  same  individuals  were  involved  as  in  the  above 
recommendation,  the  lack  of  continuity  of  vision  between  the  project 
designers -and  the  project  implementors  was  also  seen  as  a drawback. 

A number  of  other  recommendations  were  advanced  by  individual  respondents  about  what 
might  have  occurred  to  improve  project  management.  These  can  be  grouped  as  follows: 

Recommendations  about  project  management 

Integrate  concepts  sooner  into  other  subject  areas 
Increase  number  of  classrooms  involved 
Increase  interaction  time  for  teachers 


Recommendations  about  the  math  component 
More  consistency  of  vision 

Look  at  a few  quality  solutions  rather  than  a class  set 

Review  math  problems  for  ambiguity,  appropriateness  of  level  of  complexity, 

sequence  from  simple  to  complex 

Have  students  develop  math  problems  themselves 

Support  teachers  in  developing  anecdotal  reporting  skills 

Support  teachers  in  developing  consistency  when  marking 

Support  teachers  in  making  judgements  which  are  based  on  agreed-upon 

criteria 

There  were  no  recommendations  advanced  to  improve  the  management  of  the  art 
component  as  the  specialists  were  pleased  with  the  process. 

An  analysis  by  employer  group  did  not  reveal  any  significant  differences  of  opinion  although 
views  were  influenced  by  internal  issues.  The  representative  from  Alberta  Education  wished 
there  could  have  been  more  of  a focus  on  the  provincial  goal  of  building  a set  of  indicators 
for  widespread  use.  There  was  a tendency  during  the  project  to  lose  sight  of  the  provincial 
context  * 


33  What  happened  as  a result  of  the  EQI  project  that  you  did  not  expect? 

The  surprises  respondents  experienced  tended  to  cluster  around  either  process  issues  or 
project  outcomes. 

Unanticipated  aspects  of  the  process  included  the  following: 

The  push  for  quantitative  data,  the  later  abandonment  of  it,  and  the 
willingness  with  which  math  teachers  accepted  qualitative  assessments  (3) 
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Lack  of  teacher  consistency  in  marking  (1) 


The  ease  of  the  art  component  coming  together  and  the  difficulty  of  the  math 
component  in  doing  likewise  (1) 

Unanticipated  aspects  related  to  project  outcomes  included  the  following: 

The  journals  and  the  richness  of  the  information  they  provided  (3) 

New  appreciation  for  students  as  artists  or  collaborators  (2) 

New  ways  for  students  to  critique  their  work  and  talk  to  their  peers  (1) 

Increased  diversity  of  student  artwork  (1) 

The  impact  of  using  language  instead  of  letter  grades  in  math  (1) 

The  solidarity  of  findings  between  art  and  math  (i.e.,  more  generalizable 
characteristics)  (1) 

The  lack  of  correlation  between  the  dispositions  and  achievement  (1) 

Other  administrative  surprises  have  already  been  referred  to  in  3.1  and  3.2. 

An  analysis  by  employer  group  did  not  reveal  any  differences  in  opinions  overall  between 
the  two  boards. 


3.4  What  impact  will  the  EQI  project  have  on  education  in  Alberta? 

Respondents*  views  varied  about  project  impacts.  One  voice  indicated  that  the  project 
would  not  have  a provincial  impact: 

No.  A lot  of  money  was  put  into  it  and  not  to  come  out  with  something  clear  is 
bothersome.  ...  We  need  to  highlight  things  we  learned  about  student  thinking  and 
teUcher  thinking. 

The  largest  number  of  comments  were  positive  in  nature.  Typical  examples  are  as  follows: 

Hopefully  the  project  had  influenced  the  assessment  model  in  Alberta  It  has  helped  to 
break  down  barriers,  to  bring  people  out  of  isolation,  and  to  raise  their  understanding  of 
the  interaction  of  visual  and  language. 

When  the  student  can  articulate  why  their  work  is  good,  then  the  next  time  they  can  work 
on  it  and  advance  it.  There  was  a consensus  about  vision  that  made  the  project 
successful,  the  fact  that  we  were  able  to  define  quality  and  its  factors. 
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A lot  of  teachers  said  they  really  learned,  that  they  got  to  know  their  students  better  as 
individual  learners. 

It  was  an  enjoyable  project.  We  have  learned  a lot.  There  was  excellent  assistance  from 
the  teams.  They  read  a lot  on  assessment  and  caused  us  to  network  with  people  from 
other  school  systems.  I wish  we  had  another  year. 

It  had  a marvellous  empowerment  effect  on  teachers. 

It  was  a wonderful  opportunity  for  school  system  collaboration. 


Other  comments  related  to  whether  or  not  the  products  which  were  produced  were 
worthwhile.  Some  were  convinced  of  their  value: 

It  has  already  had  a substantive  impact  in  the  talk  in  school  jurisdictions. . . .Some  new 
language  emerged  from  this  project,  new  demonstrated  proof  that  teachers  are  reliable 
knowers  and  valuable  information  on  student-teacher  relationships. 

This  study  gives  people  a tangible  example  of  the  theory.  We  have  a product  and  have 
practised  applying  it  in  concrete  situations.  It  gives  people  some  starting  ground  to  start 
their  own  performance  assessment. 

We  are  giving  them  tools  and  exemplars  to  use.  The  handbook  wiU  make  teachers  more 
reflective.  Samples  of  art  have  been  judged  excellent  by  those  who  know.  In  math,  kids 
thought  in  their  heads  and  wrote  down  the  answer.  Now  we  are  asking  them  to  write 
about  the  process. 


But  others  had  a more  qualified  view: 

It  depends.  If  they  produce  the  book,  if  they  produce  a video,  if  they  incorporate  it  into 
a curriculum  package,  it  will  have  an  impact.  It  had  an  impact  on  all  the  teachers  who 
were  involved.  Teachers  who  used  it  could  substitute,  add  their  own  exemplars  and 
collect  their  own  students  work.  Students  liked  to  see  what  students  before  and  after 
them  (Le.,  younger  and  older)  could  do. 

It  depends  on  how  user  friendly  the  final  products  are.  The  teachers  have  so  marry 
choices  that  it  has  to  be  exceptional  material  It  is  a quality  project  but  it  may  be 
overwhelming  for  teachers,  particularly  at  the  elementary  level  where  they  have  7 - 8 
subjects  to  prepare  for  every  day.  The  art  EQI  handbook  is  pretty  overwhelming.  It  is 
impressive  but  intimidating.  I luzve  a subtle  caution  at  the  back  of  my  mind  that 
sometimes  teachers  want  something  simpler. 
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Others  saw  the  project  as  just  one  step  in  a long  progression  towards  performance 
assessment: 

Once  the  report  is  published  it  will  be  one  more  piece  of  evidence  to  support  the  thrust 
towards  changing  the  delivery  of  mathematics  in  our  schools.  We  must  change  it.  The 
project  will  support  movement  about  how  we  assess  students  and  I would  like  it  to  open 
the  door  to  dialogue  about  math  and  its  place  in  education  as  more  than  the  third  R. 
It  is  as  important  to  be  literate  in  math  as  it  is  to  be  literate  in  language. 

I would  hope  it  would  lend  support  to  those  in  assessment  for  valuing  the  process  and 
the  solution  as  opposed  to  the  answer.  Lots  of  people  are  pressing  to  look  at  this.  It 
becomes  a piece  of  good  information  that  we  consider.  There  is  lots  of  neat  stuff 
around.  This  adds  to  the  value  in  math  learning.  We  won*t  be  doing  workshops  on  EQI 
but  we  can't  help  but  mention  it  if  we  are  talking  about  assessment. 

We  are  moving  in  the  direction  of  performance  assessment 

However,  the  EQI  project  was  also  seen  as  a positive  influence  on  the  acceptance  of 
qualitative  research  in  Alberta: 

We  even  had  an  influence  on  Alberta  Education  staff.  They  wanted  numbers  and 
research  support,  fihat  we  were  doing  was  much  more  subjective  and  powerful  . . . 
The  way  we  did  it  was  more  of  an  action  research  approach.  The  qualitative  research 
is  legitimate  and  people  thought  about  it  I would  hope  it  would  lend  support  to  those 
in  assessment  for  valuing  the  process  and  the  solution  as  opposed  to  the  answer. 

It  is  not  fast  enough  for  me  but  it  is  having  a major  impact  on  valuing  naturalistic 
inquiry. 
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3.0  CONCLUSIONS 


Respondents  at  the  local  level  felt  that  the  EQI  project  had  resulted  in  several  useful 
concepts  in  both  art  and  mathematics  which  reflect  emerging  thought  in  the  area  of 
performance  assessment.  The  art  concepts  and  visuals  were  seen  to  be  very  useful  in  their 
current  format  while  the  math  concepts  were  seen  more  as  a jumping  off  point  for  further 
development.  Again,  due  to  the  greater  advancement  which  had  been  possible  in  the  art 
component,  the  handbook  was  seen  as  a useful  tool  for  discussions  with  parents,  while  the 
math  concepts  were  felt  to  need  further  development  before  they  could  be  discussed  with 
parents.  However,  it  was  felt  that  the  dispositions  identified  and  the  idea  of  exploring  how 
students  arrived  at  solutions  was  already  helping  teachers  understand  their  students  better. 
Respondents  agreed  that  EQI  concepts  and  the  EQI  project  process  could  be  transferred 
to  other  subject  areas.  Assessment  in  math  and  science  was  seen  to  be  linked  as  it  was  in 
art,  language  arts  and  social  studies.  The  potential  for  assessing  cross-subject  assignments 
was  seen  as  another  application.  None  of  the  respondents  had  developed  a model  to  depict 
EQI  concepts.  Perhaps  this  is  not  surprising  as  these  concepts  were  neither  linear  nor 
sequential  in  nature  but  were  rather  interactive  and  evolutionary. 

A number  of  useful  perspectives  were  advanced  with  regard  to  project  management.  The 
director  was  commended  for  the  excellent  management  of  the  project  The  resources, 
support  and  time  allotted  to  the  project  were  judged  to  be  appropriate.  Areas  which  were 
problematic  included  the  turnover  of  mathematics  personnel  on  the  project  the  lack  of 
involvement  of  senior  level  administrators  at  both  boards,  and  the  lack  of  involvement  of 
most  of  the  project  initiators  in  its  implementation.  The  committee  structure  and 
membership  was  not  clearly  understood  by  a number  of  project  participants.  The  steering 
committee  met  infrequently,  although  some  policy  direction  was  provided  at  that  level.  The 
community  advisory  committee  acted  as  a sounding  board  for  the  project  director.  It  was 
at  the  subject  area  advisory  committees  that  the  action  occurred-most  project  decisions 
were  made  here  and  implementation  of  project  concepts  in  schools  was  directed  from  these 
groups.  In  addition,  these  advisory  committees  provided  the  vehicle  for  the  networking 
which  was  viewed  so  positively  by  all  involved.  The  partnership  between  the  two  boards  was 
seen  as  very  positive  in  that  it  also  encouraged  new  networks  and  linkages.  The  project  was 
described  as  an  excellent  professional  development  activity  for  all  involved. 

Recommendations  to  improve  project  utility  in  the  art  component  related  to  upgrading  the 
binder  fotmat  of  the  handbook  and  developing  videos  of  classroom  use  to  accompany  the 
handbook.  In  the  math  component,  it  was  generally  felt  that  the  concepts  explored  and 
developed  during  this  project  should  provide  the  basis  for  additional  research  in  the  future. 
As  one  respondent  commented,  "We  need  another  year."  General  recommendations  related 
to  ensuring  that  project  findings  would  be  implemented  at  the  Calgary  Board  of  Education 
despite  the  reorganization  planned  in  the  curriculum  area  next  year,  and  to  providing 
appropriate  in-service  opportunities  for  teachers  on  EQI  concepts.  Respondents  felt  that 
project  management  would  have  been  facilitated  both  at  the  start-up  phase  and  for  the 
institutionalization  of  project  concepts  at  its  conclusion  if  senior  administrators,  project 
initiators  and  the  CBE  assessment  team  had  been  involved.  Unanticipated  outcomes 
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identified  by  respondents  related  mainly  to  the  unexpeaed  richness  of  findings  and  their 
utility  and  transferability,  as  well  as  to  a new  appreciation  for  students  as  collaborators  in 
the  assessment  process.  Overall,  it  was  hoped  that  the  EQI  project  would  have  a positive 
impact  on  performance  assessment  praaices  in  Alberta  in  general  and  in  the  Calgary  area 
in  particular.  It  was  also  evident  that  this  would  take  time  to  accomplish. 
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